MOMENT MAPS AND RIEMANNIAN SYMMETRIC PAIRS 



0^ 



LUIS O'SHEA AND REYER SJAMAAR* 



> 

o 
in 

(N 

o 



Abstract. We study Hamiltonian actions of a compact Lie group on a sym- 
plectic manifold in the presence of an involution on the group and an anti- 
symplectic involution on the manifold. The fixed-point set of the involution 
on the manifold is a Lagrangian submanifold. We investigate its image under 
the moment map and conclude that the intersection with the Weyl chamber is 
an easily described subpolytope of the Kirwan polytope. Of special interest is 
the integral Kahler case, where much stronger results hold. In particular, we 
obtain convexity theorems for closures of orbits of the noncompact dual group 
(in the sense of the theory of symmetric pairs). In the abelian case these re- 
sults were obtained earlier by Duistermaat. We derive explicit inequalities for 
polytopes associated with real flag varieties. 
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1. Introduction 

The main result of this paper is a "real form" of Kirwan's convexity theorem. We 
apply our result to flag varieties of real semisimple groups and thus obtain eigen- 
value inequalities, which generalize inequalities found by Weyl, Ky Fan, Kostant, 
Klyachko, and many others. For instance, we find an answer to the following ques- 
tion: given the singular values of two rectangular matrices (of the same size), what 
are the possible singular values of their sum? 
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2 LUIS O'SHEA AND REYER SJAMAAR 

We obtain these results by studying symplectic manifolds M equipped with an 
antisymplectic involution r and a Hamiltonian action of a compact Lie group U . 
We assume the fixed-point set of the involution to be nonempty, in which case it is a 
Lagrangian submanifold. We denote the moment map for the [/-action on M by $. 
Our goal is to describe the image of the Lagrangian under $. Good results can of 
course only be expected if the involution and the action are related in a reasonable 
way. The right condition to impose is for $ to be "anti-equivariant" relative to the 
involution r and an involutive automorphism a on the group U . The Lagrangian 
is then not stable under the action of C/, but only under the fixed-point group Lf^ . 
Thus the theory of Riemannian symmetric pairs comes into play. 

If t/ is a torus and a is inversion, anti-equivariance amounts to mvariance of <& 
with respect to r. This special case was studied by Duistermaat B, who proved 
that the image of AH under $ is equal to the full image $(Af ). 

This turns out to be false in general. Our main result is the following description 
of $(Af '^) as a subset of $(M). Instead of the image $(M), it is easier to consider 
the intersection A(M) = $(M)nt!^, where t!^. is a Weyl chamber in u*. According to 
Kirwan, A(M) is a convex polytope, called the moment polytope of M . Our result 
is that the set A(M'^) = (^{Nr)CM\ is equal to the subpolytope of A(M) consisting 
of symmetric elements. In other words, A(M'^) is equal to the intersection of A(M) 
with the Weyl chamber of the restricted root system. 

The proof is an adaptation of the proof of Kirwan's convexity theorem given in 



28[ . (We also attempted to adapt the perhaps simpler approach followed by Lerman 
et al. [gl[, but were unable to do so.) Briefly, the method is to establish the result 
first in the Kahler case (more precisely, for projective and afline varieties), and 
then handle the general case by dint of a local normal form theorem, which asserts 
that near the orbits that matter every Hamiltonian [/-manifold is isomorphic to an 
affine variety in a suitable sense. An advantage of this approach is that it yields 
a characterization of the vertices and an attractive, if unwieldy, invariant-theoretic 
local description of the polytope near any point. More importantly, the Kahler case 
has the noteworthy feature that G, the noncompact real reductive group dual to 
U , acts holomorphically and commutes with the antiholomorphic involution r. We 
are thus led to a convexity theorem for closures of G-orbits, which is a nonabelian 
version of a result of Atiyah's. 

In Section we introduce the category of Hamiltonian ([/, o')-manifolds and give 
a number of examples. Our main result is stated in Section |3|. Section ^ is an 
expose of Kahler quantization for Hamiltonian (C/, cr)-manifolds, where we develop 
the techniques needed for establishing the convexity theorem in the Kahler case, 
which we carry out in Sections |5| and g. We then state the normal form theorems 
in Section [7| and finish the proof in Section ^ Applications inspired by mechanics 
(cotangent bundles) and representation theory (coadjoint orbits) are discussed in 
Section y. A few elementary results on antisymplectic maps and symmetric pairs 
are covered in Appendices |A| and ^. Appendix |C| is a list of notational conventions. 

2. Elementary properties and examples 

Let [/ be a compact Lie group. Let a be an involution of [/, i.e. an automorphism 
which satisfies a^ = idjj- (We are mostly interested in the case where a is not the 
identity.) The derivative of a induces involutions on the Lie algebra u and on the 
dual space u*, both of which we denote for simplicity also by a. 
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2.1. Definition. A Hamiltonian {U, a) -manifold is a quadruple {M,lu,t,^) con- 
sisting of a smooth manifold M equipped with an action of the group U, a sym- 
plectic form co, an antisymplectic involution t (i.e. a diffeomorphism r satisfying 
T^ = idM and t*lu — — w), and a [/-equivariant map $ : M ^ u* which is a moment 
map for the action of \J . In addition we impose the following two conditions: 

^{T{m)) =-(T($(m)), (2.1) 

T{um) — ij{u)T{m) (2-2) 

for all TO g M and u G U. 

Consider the semidirect product U x: {±1}, where the action of {±1} on U is given 
by the involution a. (By {±1} we mean a copy of Z/2 written multiplicatively.) 



Then (2.2) says that U >^ {±1} acts on M and that the action is sesquisymplectic 
in the sense that u*u! = e{u)u!, where e: [/ x {±1} -^ {±1} denotes the canonical 
homomorphism . 

Properties {2A) and (2^) are not independent. For ^ e u we denote by ^m 
the induced vector field on M and by $^ the function on M defined by ^^{m) = 
($(to),0. 



2.2. Lemma. If U is connected, then (2.1) implies (p.2[). Conversely, if (2.2) is 



satisfied, we can shift ^ by a suitable constant to ensure that (2.1) holds 



Proof. Using the moment map condition (i$^ = i{£,M)oJ we obtain from (2.1) that 



r.i^M) - (ff(O)M (2-3) 



for all ^ e u. This implies (2^) if C/ is connected. Conversely, if (2_^) holds then 



( p.3| ) holds, whether or not U is connected. Therefore 

(d$,0 = i(6-/)^ = t(r,cr(C)M)w = -l{t^c7{^)m)t*uj = -r* (t(cr(^)M)cj) 

= -r*(d$,a(0)=-(rf($oT),a(0)--(rf(ao$oT),C) 

for all ^ and so d((7 o $ o r) = — (i$. Furthermore, it is easy to check that cr o $ o r 
is f7-equi variant. It follows that 

(TO $0 T = -<i> + c 

for some [/-invariant c G u*. Multiplying by a on the left and by r on the right 
we get $ = — (7 o $ o T + cr(c), and therefore it(c) = c. Putting <!>' = ^ + c', with 
c' = — c/2, we see that $' is a {/-equivariant moment map satisfying 

CrO<I> OT = (To(c()-fc)oT==(TO<(>Or + C = — $ + c+ c = — $ , 

i.e. $' o T = -cr o (()'. D 

Henceforth we assume U to be connected, apart from a few exceptional situations 
where for technical reasons we need to allow disconnected groups. These exceptions 
will be pointed out explicitly. 

The focus of our attention is the fixed-point manifold M"^. Let K be the identity 
component of the fixed-point group W^ and let q C u be the — 1-eigenspace of cr. 
Then u = 6 © q, where i — Lie{K). Identifying i* with the annihilator of q and 
q* with the annihilator of 6, we have u* = t* ® q*. Choose an almost complex 
structure J on M which is [/-invariant and r-anti-invariant. The existence of such 
a J is guaranteed by Lemma A.3(^) in Appendix ^. 
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2.3. Proposition. (i) // Af^ is nonempty, it is a V^ -stable totally real La- 
grangian submanifold of M, and $(Af'^) is contained in q*. 
(ii) Every connected component of M'^ is K -stable. 
(iii) The K -orbit through every m in M'^ is a connected component ofUmD M'^ . 



Proof. The inclusion $(A/'^) C q* is obvious from (|2.l|) and the [/'^-stabihty of Af^ 
is obvio us fr om ( |2.2|) . The fact that Af^ is Lagrangian and totally real follows from 
Lemma [A^ (|ii|) (apphed to the group Q = U yi {±1})- This proves (|). 

^ follows immediately from (m) and the fact that K is connected. 

Let m e Af^. It follows from (||) that Km C Uraf] M'^ . To prove (|ij) it suffices 
to show that Km is open in UmDM'^, or equivalently that the tangent space to Km 
at m is equal to Tm{Um) n T^M'^. A general tangent vector to Um at m is of the 
form ^M = Cm + Cm with ^+ e i and ^- e q. Then t^^m) = ct(C)m == Cm - Cm' 
so £,M is tangent to Af"^ if and only if t*(Ca/) = Ca/ if and only if Cm = 0' i-^- 
^M=^ti^Trn{Km). D 

2.4. Example (tori). Let U he a torus and a the involution which sends u to u~^. 



Then f/'^ is the subgroup of 2-torsion elements of U and condition (2.1) boils down 



to $ being invariant under r. This is the setting of Duistermaat's paper H. 

2.5. Example (connections). This infinite-dimensional example was suggested to 
us by C. Woodward. Consider an oriented Riemann surface S and an orientation- 
reversing involution / : S ^ E without fixed points. Let P be a trivial principal 
{/-bundle over S and 21 the space of fiat connections on P. According to Atiyah 
and Bott 21 is a Hamiltonian il-manifold, where il = C°°(S,C/) is the group of 
gauge transformations. Involutions on 21 and il are given by puUback: t{A) = f*A 
and <y{u) — f*u. Then r is antisymplectic because / reverses the orientation and 
(T is a group homomorphism. One checks easily that 21 is a Hamiltonian (U, cr)- 
manifold. Moreover, the Lagrangian 21"^ is isomorphic to the space of connections 
on the quotient bundle P/ f over the non-orientable quotient surface E//, and il*^ 
is the gauge group C°°{T,/f, U). 

2.6. Example (cotangent bundles). Let S be any manifold with an involution r 
and let M — T*She the cotangent bundle of 5", equipped with its standard symplec- 
tic form. The cotangent lift of r is a symplectic involution of M. We turn it into an 
antisymplectic involution, also called r, by "reversing momenta" , i.e. by multiplying 
by —1 in the cotangent directions. The fixed-point set AP^ is equal to the conormal 
bundle of S'^ . Suppose that U acts on S in such a way that t{us) = a(u)T{s) for 
s € S, and lift the action to M in the canonical way. It is then a straightforward ex- 
ercise to check that Af is a Hamiltonian ([/, cr)-manifold. The moment map is given 
by $^(s,p) — _p(Cs,s) for C G u, s G 5" and p € T*S. For instance, we can take S to 
be the group U with involution a. If we identify M = T*U with U xu* hy means of 
left translations, then r is given by t{u, C) = (f (w), —(yiS,)) j and Af^ == U"' x q*. On 
A/ there are three different Hamiltonian [/-actions, namely the actions induced by 
left multiplication, right multiplication, and conjugation. Thus A^ is a Hamiltonian 
([/, (T)-manifold in three different ways. 

2.7. Example (vector spaces). In this example U need not be connected. Let 
{V,uj) be a finite-dimensional symplectic vector space and let r e GL(V) be an 
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antisymplectic involution. Assume that U acts linearly and symplectically on V 



and that (2.2) holds for all u € U and m £ V. Let $y be the standard quadratic 



moment map for the tZ-action given by 



<^U^) = Ilo{^v,v). (2.4) 



Then the quadruple (y,a;,T, $y) is a Hamiltonian ([/, cr)-manifold, i.e. (2.1) holds 



automatically. The fixed-point set V^ is clearly nonempty and is in fact a W^- 



stable Lagrangian subspace. Furthermore, by Lemma A.l there exists a [/-invariant 
and T-anti-invariant compatible complex structure J on V. Then we can define a 
real inner product f3 on V hy [3{v,w) — uj{v,Jw) and a Hermitian inner product 
h hy h — f3 + ilo. Clearly h is [/-invariant and h{T{v),T{w)) = h{w,v). As a 
unitary [/"'-module (but not as a [/-module) V is canonically isomorphic to the 
complexification of V^ ^ and r is complex conjugation relative to the real form V^ . 
Let yl: [/ — > V{V) be the homomorphism defining the action of U on V. We 
call the data {V,uj,t,J,A) (often abbreviated to V) a unitary (U, a) -module. See 
Section || for a geometric construction and a classification of such modules. 

2.8. Remark. Let y be a unitary ([/, CT)-module. Consider the unitary group 
U(V^) and the standard involution on V{V) defined by cry(a;) — {x^^Y, where "i" 



denotes transpose with respect to the Hermitian form h. It follows from (2^) that 
the representation A intertwines the involutions on U and \J(V): Aa = ay A. In 
fact, this property can be taken as an alternative definition of a unitary ([/, a)- 
module. Here is yet another, purely complex algebraic characterization. Define 

h{v,w) = h{v,T{w)). (2.5) 

Then 6 is a nondegenerate symmetric complex-bilinear form and is "twisted" in- 
variant under [/ in the sense that b{uv,a{u)w) — b{v,w) for all u G U and v, 
w €z V. Conversely, a [/-module furnished with suc h a form can be endowed with 
the structure of a unitary ([/, (T)-module such that (^.5[) holds. 



2.9. Example (coadjoint orbits). Let A e u* and let M = UX be the coadjoint 
orbit through A, equipped with its standard symplectic form. Assume that — cr(A) G 
M, so that — cr(M) — M. Then we can define an involution r on M by putting 
t(x) — —a{x); this is antisymplectic because ct is a Lie algebra homomorphism. 



The moment map is simply the inclusion of M into u*, and therefore {2A) holds 
by construction. Note that the fixed-point set Af^ is equal to M n q*, so if it is 
nonempty wc may assume that A is in q*. We call the coadjoint orbit M symmetric 
if it is of the form UX with A G q*. We assert that if A S q* then M"^ is a single 
iT-orbit, 

Ar = [/'"A ^ KX. (2.6) 



To prove this we invoke Proposition 2.2, according to which it is enough to show 
that Af"^ is connected. Observe that the involution on u maps [u, u] into itself. Let 
U be any connected Lie group with Lie algebra [u, u] such that the involution on 
u descends to an involution a ow U . Then U acts on u and u*, and the orbits of 
U are the same as those of U . It follows from the proof of Proposition 8.8(ii) in 
Helgason ||l|, Ch. VII] that M^ = O^X for any such U. We can take U to be the 
universal cover of [[/, [/], in which case the group U" is connected by Theorem 8.2 
of |13, Ch. VII]. The upshot is that M'^ = Lf^X is connected, which concludes the 



proof of (^.61). This example is continued in 3^, L8, 5^, 5.14 , and in Section ||, 
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2.10. Example (syniplectic quotients). Let AI be a Hamiltonian ([/, (T)-nianifold 
and let if be a Lie group which acts on M in a Hamiltonian fashion with momentum 
map ^ : Af — > t)*. Assume that the action of H commutes with that of U. Assume 
also that t maps ^^^(0) into itself and maps iJ-orbits to if -orbits, i.e. for allh G H 
and m E AI there exists h' G H such that T(hm) — h'T{rn). (A sufficient condition 
for this to hold is that H should be connected and that for every 77 G t) there should 
exist 77' such that r*(^'') = '^^ .) Assume furthermore that H acts properly and 
freely on the fibre *"^(0), so that the quotient M//H = ^~i(0)/ii is a well-defined 
symplectic manifold. The action of U and the map t descend to M //H and it is 
easy to see that Ad//H is a Hamiltonian ([/, (7)-manifold. 

2.11. Example (projective varieties). Let y be a unitary ([/, cr)-module as in Ex- 
ample 2.7. Taking H — S^ va Example ETfl and letting H act by complex scalar 



multiplication on V , we conclude that the projective space W = V// S^ is a Hamil- 
tonian [U^ (T)-manifold in a natural way. Likewise, every nonsingular complex sub- 
variety of PV^ which is stable under U and r inherits the structure of a Hamiltonian 
(f/, cr)-manifold. 



2.12. Example (toric varieties). Let (U, a) be as in Example 2.4, let A be a simple 
lattice polytope in u* , and let Af be the symplectic toric variety ( "Delzant space" ) 
associated with A. As explained e.g. in ||2^, this space can be obtained from T*U 
as a symplectic quotient with respect to an appropriate subtorus of U and therefore. 



by Examples |2.6| and 2.10 , has the structure of a Hamiltonian ([/, (T)-manifold. In 
fact, Ai is a complex projective variety in a natural way and as such is defined 
over Z and hence over M. It is not hard to see that the antisymplectic involution 
r on Af is the same as complex conjugation. The real part M'^ of Ad is nonempty 



and connected. See e.g. |15 , where the cohomology of Ai'^ with Z/2-coefficients is 
computed. 

2.13. Example. The product of a Hamiltonian (J/i, (Ti)-manifold and a Hamil- 
tonian (t/2, o'2)-manifold is a Hamiltonian {Ui x U2,(Ji x (T2) -manifold. Let H be 
a cr-stable closed subgroup of U . For simplicity we denote the restriction of a to 
H also by a. Then every Hamiltonian ([/, cr)-manifold inherits the structure of an 
(ii, (T)-manifold. In particular, the product of two Hamiltonian (t/, cr)-manifolds is 
a Hamiltonian (f/, cr)-manifold with respect to the diagonal action. Observe that 
these facts are also true if the groups in question are not connected. 

2.14. Example (duplication). Let (Ai,w, $) be any Hamiltonian [/-manifold. Let 
M~ be the same manifold as Ai, but furnished with the opposite symplectic form 
-uj. Define M'^'^^ ^ AI U Ai", the disjoint union of Af and Ai", and define r to 
be the antisymplectic involution which interchanges the two components. It has 
obviously no fixed points. Extend the [/-action and the moment map from Af to 
^dup ^jy putting uT{m) = T{a{u)Tn) and ^(T{m)) = —a{^{m)) for m e Af . Then 
Af "^"P is a Hamiltonian ([/, o')-manifold. For instance, if Af = U\ is a coadjoint 
orbit of [/, then Af ^"p = [/A U -a{UX). 

2.15. Example. The process of duplication has linear analogue, which runs as 
follows. Let V be any unitary [/-module with Hermitian metric h = P + iuj. Let V~ 
be the vector space conjugate to V equipped with the Hermitian form h = (3 — iuj. 
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Put T/'^"? = V ® V~ and let r be the map which interchanges the two components. 
It is obvious that r is conjugate hnear and antisymplectic. Turn V^ into a U- 
module by putting ut{v) = t{(j{u)v) for v €V; then {V^^^^t) becomes a unitary 



([/, cr)-module as in Example 2.7. The fixed-point set of r is the diagonal. 



Here is a general criterion for the fixed-point set A/'^ in a Hamiltonian ([/, a)- 
manifold (M, aj,T, $) to be nonempty. The integrality condition is satisfied for 



instance if M is prequantizable in the sense of Definition 4.1 



2.16. Proposition. Suppose that M is a compact connected (U, a) -manifold of 
dimension An and that there exists a class c G i?^(M, Z) such that t*c — — c 
and the image of c in H'^{M,M.) is equal to the cohomology class ofuj. Then M'^ 
is nonempty and 6"|m^ is nonzero, where b £ _ff^(Af, Z/2) denotes the reduction 
modulo 2 of c. 

Proof. Observe that t*6" — 6" and that 6^" is the mod 2 fundamental class of M. 
The result now follows from Theorem 7.4 of Bredon |§, Ch. VII]. D 

We do not know of a good criterion for the fixed-point set to be connected. 



In Examples 2.7, 2.9 and 2.12 the total space M is simply connected an d th e 



Lagrangian M'^ is connected. The same is true for M = T*U (as in Example 2.6), 
provided that U is simply connected. But here is an example where M is simply 
connected and M'^ is disconnected. Let S be the two-sphere with the involution 
T given by rotation through tt about the vertical axis, and let U ~ SO(3) act on 
S in the usual way. Then t(us) = a{u)T(s), where cr is the inner involution of U 
defined by the matrix 



-1 











-1 











1 



Let M = T*S as in Example y^. Then M'^ is the union of two Lagrangian planes. 
However, this example is noncompact. This leads to the following question. 

2.17. Problem. Is M'^ connected whenever M is compact and simply connected! 

3. The main result 

Let (M, uj, r, $) be a Hamiltonian {U, cr)-manifold. Let T be any maximal torus 
of U and t^ a (closed) fundamental Weyl chamber in t* . For any subset X of M we 
define A{X) to be the intersection $(X) nt!j_ . A well-known theorem of Kirwan [^ 
(as generalized by Hilgert, Neeb and Plank iQ and Sjamaar ^^) says that if M 
is connected and <& is proper, then A(Af) is a convex polyhedron in t*. The main 
result of this paper is a comparable assertion about the Lagrangian submanifold 

]\/r of M. 

For this result to hold, the torus T and the chamber t;^ need to be chosen in a 
suitable "position" relative to the involution a, namely as follows. As before, let K 
be the identity component of W^ and q C u the — 1-eigenspace of the involution a. 
Choose a maximal abelian subspace o of q and a maximal abelian subalgebra t of u 
which contains a. Let A = exp a and T = exp t be the corresponding subtori of U. 
Let R be the root system of (C/, T), R' C R the subsystem of roots which vanish 
on a, and R° the restricted root system. (See Appendix P.) Fix sets of positive 
roots R'_^ in R' and i?^ in R°. Define the set of positive roots R+ in R to be the 
union of i?^ and those roots in R which restrict to elements of i?!}.. Denote the 
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corresponding fundamental Weyl chambers in t* and a* by a*^_ and t^J. , respectively. 
Then by Lemma |B.2| , t!^ n q* = t;^ n a* is equal to a!j_. 



Now consider the iiT-stable subset ^{M'^) of u* and the intersection 

A(M^) = $(iir ) n t;. 

Since $(M'^) C q*, A(Af^) is equal to $(Af^)na!j_. Our main result is the following 
convexity theorem for M'^ . 

3.1. Theorem. Assume that M is connected, M'^ is nonempty, and the moment 
map $ is proper. 

(i) A(M'^) is equal to A(A/)na* and is therefore a closed convex polyhedral subset 

of a*. 
(ii) IfXeaXisa vertex of A{Ar) and m G $"i(A)n Af^, then A C [Ux, Ux] [/„. 
In particular, the vertices of A{M'^) which are contained in the relative inte- 
rior of V^ are images under <I> of A- fixed points in Af^. 

We call A(Af^) the moment polyhedron of Af^. From (|) it follows easily that 
$(M'^) = $(Af) n q*. Also, (|) can be interpreted as saying that if a complete 
list of inequalities for the polyhedron A(Af) is known, then the inequalities for 
A(Af '^) can be obtained simply by adding to the list a number of equalities, namely 
those describing the subspace a*. However, as we shall see in Section g, this is an 
inefficient method for finding the inequalities for A(Af'^). 

In Sections ra and ^ we prove the theorem for projective and affine varieties. 
The general case is treated in Section |[ where we reduce it to the algebraic case 
by means of a local normal form theorem. The algebraic versions of Theore m p?^ 
enable us to generalize it to various situations where M is singular (Theorem ^.10 



and its corollaries), or where <& is not proper (Theorems |6 . 3| and 9.1). We also prove 
a result. Theorem |8.2| , which describes, for any m G Af^ mapping into A(Af'^), the 
shape of the moment polyhedron near $(?7i) in terms of the coordinate ring of an 
affine variety determined by infinitesimal data at to. 

3.2. Example. If a is the identity map on U, then a = {0}. The Lagrangian Af"^ 



is [/-invariant and satisfies <I>(Af '^) = {0}, so Theorem 3.1 is trivially true 



3.3. Example. If M is compact and C/ is a torus, Theorem B.l says that A{M'^) 
is the convex hull of $(Af^nA4"^). If in addition a{u) := u'^, then $(Af^) = $(Af), 
which is the main result of Duistermaat's paper &. 

3.4. Example. Let A £ a!^ = t^i n q* and let M be the symmetric coadjoint 



orbit UX of Example 2.£. T hen Af^ ~ KX. Consider the action of the maximal 
torus T on M. By Theorem lO, A(A/^) = hull(Af'^ n Af). According to Lemma 



B.4K |ii|), M'^dM^ is equal to W°X, the restricted Weyl group orbit through A. The 
conclusion is that A{M'^) = hullVF"A. This was proved by Kostant in ^^. Here 
is a slightly different way of recovering the same result: Kostant also proved that 



A(Af) = huUT^A. Therefore, by Theorem |3T| and Lemma |r|(|]), 



A(Ar) = A(A/) n a* = (hullT4^A) Ha* = hullTy"A. 
See Section || for yet another proof. 

We emphasize that Af^ is not stable under the action of the torus A. Duistermaat 
has given a more intrinsic description of the set M^ n Af^ , which does not refer to 
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the action of A on the ambient manifold M. The submanifold M'^ is stable under 
the 2-torsion subgroup A[2] = A" , and clearly M^ n W is included in (M'^)^!^]^ 
This inclusion is often, but not always, an equality. (For an example where it is 
not, let [/ = SU(2), a = complex conjugation, so that V^ — K = SO(2) and 
A = T. Let M be a coadjoint [/-orbit. Then M is a sphere, M"^ is a great circle, 
M^ n A-r = M'^ consists of the two poles, but A[2] is the centre of C/, so it acts 
trivially on M .) To remedy this situation, define the sequence of submanifolds Yfc 
of M^ by Yfc = M-^I^ ' nM^ for fc > 0, where A[2^] denotes the 2*''-torsion subgroup 

of A. One easily proves that ^[2*^] maps Y^-i into itself and thus that Yfc = Y^_-^ ' 
for k > 1. Because {Yfe} is a nested sequence of submanifolds, it is eventually 
constant. Since lj/c>i ^[2*^] is dense in A, we obtain M"^ n AH = Yi for sufhciently 
large I. (Cf. ||g. Lemma 3.3].) 

4. Geometric quantization 

The proof of Theorem |3.l| in the algebraic case makes use of geometric quantiza- 
tion. Given a Hamiltonian ([/, cr)-manifold M which admits a Kahler polarization, 
how is the involution on Ad reflected in the quantization Q{AA)1 In this section 
we show that, subject to a compatibility condition on the involution and the po- 
larization, Q{A'I) is a unitary (/7, cr)-module as in Example ^.7\ The compatibility 
condition is for example satisfied for a class of coadjoint orbits which contains, but 
in some cases is strictly larger than, the class of symmetric coadjoint orbits. For 
such an orbit the quantization is a unitary ([/, (7)-module which is irreducible as 
a [/-module. As a byproduct we obtain a characterization in terms of the orbit 
method of representations of real and quaternionic type, and a classification of 
unitary (C/, o')-modules. 

The details are as follows. Let (M, uj, r, $) be a Hamiltonian (C/, o')-manifold. 
Suppose that the symplectic form is integral. Let L be a prequantum line bun- 
dle, that is a complex line bundle with Hermitian fibre metric h and Hermitian 
connection V with curvature —2Triuj. 

4.1. Definition. The prequantum bundle L is {U,a)-equivariant if there exist a 
lifting of the [/-action on Ad to a unitary action on L and a lifting of r to a real 
linear bundle map t^: L -^ L which is conjugate linear (tl(cI) = ctl{1) for all 
I S L and c G C), maps horizontal subspaces to horizontal subspaces, and is an 
involution (r| = id^). 

The lifting tl defines an isomorphism (L, V) -^ (L, V) of complex line bundles 
with connection and therefore induces an an equivalence (T*L,r*V) = (L, V). 

The vector fields generating the [/-actions on M and L are related by Kostant's 
formula 

a - (^f )hor + 27r$«l/i (4.1) 

for ^ € u. (See e.g. jl^, |l^.) Here "hor" refers to the horizontal lift of a vector field 
and vl denotes the infinitesimal generator of the scalar S'^-action on L. This shows 
that the obstruction to lifting the [/-action is purely topological and that the lift is 
unique, if it exists. 

Let us now analyse the condition on r. Because r is antisymplectic, the curvature 
of (r*L, r* V) is equal to that of (Z, V), and therefore L' — t*L L* is a bundle 
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with flat connection V' = r* V® 1 + 1 ® V*. Hence, according to the exact sequence 

H\M,Z) = — > H\M,R) — > H\M,S^) — > H^{M,Z) — > H^{M,R) — > ■■■ , 

the pair (L',V') is represented up to equivalence by a class in H^{M, S^). We 
conclude that a connection preserving conjugate linear lifting tl of r exists if and 
only if this class vanishes. According to Proposition 1.12.2 of Kostant [|l9[, any 
another such lifting r^ is given by t'j^ — ctl for some nonzero complex number c. 
Moreover, we can choose c such that r^ maps h to h, that is to say, preserves the 
real inner product Re h. Such a lifting is "almost" an involution. 

4.2. Lemma. Let tl be conjugate linear lifting of t which preserves the connec- 
tion. 

(i) If tl preserves Re ft,, then t^ = a idL for some complex number a of norm 1. 
(ii) If tl is an involution, then tl preserves Re/i. 
(iii) Let r^ be another conjugate linear lifting of t which preserves the connection. 

If Tl is an involution, then r^ is an involution if and only if it is of the form 

t'j^ = ctl with |c| = 1. 

Proof. Assume that tl preserves Re h. Since r is an involution, the map r| is then 
a (complex linear) automorphism of L which covers the identity and preserves V 
and h. It follows from the aforementioned result of Kostant 's that t1 — a id^, for 
some a in the unit circle. This proves mj. 

Again by Kostant's result, the fact that tl is conjugate linear implies that r£ft, = 
bh for some b > 0, and hence r^Keh = 6 Re ft. If tl is an involution, then Reft = 
(t£)^ Re ft = 6^ Re ft, so 5 = 1 and r£ Re ft = Re ft. This proves ([]). 

Finally, assume that r^ is an involution. We know that t£ = ct^ for some c ^ 0. 
Therefore r£ is an involution if and only if id^ = (t£)^ = ct^ctl ~ cct^ = |cp id^, 
i.e. \c\ = 1. This proves (pli[). D 

4.3. Remark. Even if r is liftable, it is not always possible to flnd an involutive 
lift. In some cases one can only find a lift tl which satisfies the "quaternionic" 



condition r^(/) = —I. See also Remark 4.9 



Suppose from now on that M admits an integrable complex structure J which 
is ?7-invariant, t- anti-invariant and compatible with lo. Then cj is a positive (1, 1)- 
form, so L admits a holomorphic structure such that the projection L — j M is 
holomorphic. In addition, wc can choose the metric connection (V, ft) to be com- 
patible with the holomorphic structure. 

4.4. Lemma. Let t^: L —> L be any real linear bundle map which lifts t and 
preserves the real inner product Reft. Then tl is antiholomorphic if and only if tl 
is conjugate linear and preserves the connection. 

Proof. Assume that tl is antiholomorphic. Then the derivative of tl at any point 
in L is a conjugate linear map. By taking a point m in the zero section we see 
that the restriction of (t^)* to Lm is a conjugate linear map Lm — *■ Lrim)! which 
implies that the restriction of r^ to every fibre is conjugate linear. Let f : L -^ M. 
be the smooth function defined by f{l) = h{l,l). The fact that the connection is 
Hermitian implies that the horizontal subspace at I is equal to 

Hi = ker dif Hi ker dif. 
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Together with the fact that r^ is antiholomorphic and preserves / this iniphes that 
{tl)*'Hi = 'Hr^(i) for all I £ L, that is to say tl preserves the connection. 

Now assume that r^ is conjugate linear and preserves the connection. The 
tangent space to L at any point / decomposes as T/L = Tii (B Lm, where m is 
the base point of /. Because tl preserves the connection, its derivative at / is 
the direct sum of two maps L,„ — > L^(„j) and Hi -^ Ti^-^ij^y Both of these maps 
are conjugate linear, the first one by assumption and the second one because r 
is antiholomorphic and the connection is complex. Therefore {tl)* is conjugate 
linear, i.e. r^ is antiholomorphic. D 

Henceforth we assume L to be (C/, i7)-equivariant and fix an involutive lifting tl 



as in Definition 4.1. Then tl preserves Re/i by Lemma 4.2 and hence is antiholo- 



morphic by Lemma 4.4. Let Q{M) = T{M, L) be the space of global holomorphic 
sections of L. We define an involution on the global smooth sections of L by 
sending a section s to tl o s o r. Because r and tl are both antiholomorphic, 
this involution maps Q{M) into itself. We denote its restriction to Q{M) by tq. 
Since tl is conjugate linear, so is tq. Moreover, tq preserves the real part of the 
Hermitian inner product on Q{M) given by {s,t) — Jj^ h(^s{m),t{m)^ dm, where 
dm — \uj"'/nl\ is Liouville measure on M. With this inner product, Q{M) is also 
a unitary tZ-module in a natural way. The involution and the [/-representation on 
Q{M) are related in the expected manner. Since r^ preserves the connection, we 



have (rL)*Xhor — (T*X)hor for all tangent vectors X. Together with (2.1), (p.3| 



and (4.1) this implies that for all ^ G u 

{TL)*iL = (TL)*((6/)hor) +27r($«OT)(rL),Z.L 
= (T4M)hor-27r$-(«)(-Z.i) 



^iO 



L- 



Therefore tl{uI) — a{u)TL{l) for all u G C/ and I G L. From this we obtain easily 
that tq(us) = cr(u)TQ{s) for all u £ U and holomorphic sections s. The result is as 
follows. 

4.5. Proposition. The vector space Q{M) is a unitary (U,cr)-module as in Ex- 



ample 2.7, finite- dimensional if M is compact. Two different involutive connection- 



preserving liftings of t give rise to isomorphic unitary (U, a) -modules. 



Proof. It remains to prove the uniqueness part. According to Lemma 4.2, any other 
lifting t'j^ with the stated properties is of the form t'j^ = ctl with |c| = 1. Then t'j^ 
gives rise to the involution Tq — ctq of Q{M). The scalar multiplication operator 
C{s) = -^/cs is unitary, commutes with U and satisfies CtqC~^ = C^tq — Tq, 
so defines a unitary intertwining operator between the (C/, cr)-modules \Q{M),tq^ 
and (Q(M),T^). D 

Kodaira's Embedding Theorem implies that if M is compact, it is biholomor- 



phically equivalent to a complex projective variety. Using Proposition 4.5 one can 
easily show that Kodaira's map, j, is C/ x {±l}-equivariant, and hence that its im- 
age j{M) is a [/-stable and r-stable nonsingular projective variety, as in Example 



2.11 . In general j is not a symplectomorphism, but a straightforward Darboux-type 
argument shows that there exists a C/ x: {±l}-equivariant diffeomorphism / from 
M to itself such that j o / is a symplectomorphism. The conclusion is as follows. 
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4.6. Corollary. Every compact Hamiltonian (U,cr)- manifold admitting a {U,cr)- 
equivariant Kdhler prequantization is symplectically and {U,a)-equivariantly diffeo- 
m,orphic to a complex projective (U, a) -manifold. D 



Using Proposition 4.5 we can give a complete description of all finite-dimensional 
(f/, (T)-niodules. Let us first investigate the irreducible case. It is well-known that 
the Kahler quantization of a Hamiltonian [/-manifold is an irreducible [/-module if 
and only if the manifold is isomorphic to an integral coadjoint orbit, i.e. an orbit of 
the form UX with A integral dominant. Such an orbit is diffeomorpic in a natural 
way to the complex homogeneous space U^/Px, where U^ is the complexification 
of U and P\ is the parabolic subgroup associated with A. It follows from the 
Borel-Weil Theorem that the quantization of UX is the irreducible module V\ with 
highest weight A. We shall establish a necessary and sufficient criterion for Vx to 
be a ([/, cr)-module. 

First we need an auxiliary result. Let K' be the centralizer oi A in K. Then 
T' = {K')o n T is a maximal torus of K' . Let W <Z W the Weyl group of (the 
identity component of) K' with respect to T' and let w'^ be the longest element 
of W . Define a+: i* -^ i* by g- +(A) — — cr(woA). As before, i?+ denotes a set of 



positive roots in t* chosen as in (B.4) 



4.7. Lemma. (i) a^ is an involution and cr_|_(i?+) = R^. Hence a^(i*^) = i*^ 

and a^ preserves the partial ordering on the weight lattice defined by i?+. 

(ii) o'-i.(A) — X if and only if either A G a*, or A S (t')* and X — ~WqX. 

(iii) Let ko G M/K')aiT') be any representative ofwQ. Let X be an integral weight 
and let x\ denote the character ofT obtained by exponentiating X. If X€ a*, 
then XxiH) = 1- V ^ ^ (t')* "'^^ •^ = ~w'qX, then Xa(^§) = ±1- 

Proof. The action of W' fixes a* and therefore commutes with a. Since w'q and a 
are involutions, so is <j+. Observe that w'q maps i?^ to R'_ and permutes i?,+ \ R'_^, 
whereas a fixes i?^ and sends R^ \ R'_^ to R- \ R'_ . This implies that WqO a maps 
R+ to R-. Therefore a+iR+) = R+. This proves |). 

The fact that w'q fixes a* implies that a+ respects the decomposition ( |B.2| ), and 
that cr_|_|a* = id and cr-|_|(t/)* = ^w'q. This immediately implies (|i|). 

Let A be an integral weight. Since w'q is an involution, /cq is in T'. If A G a*, 
then T' is contained in the kernel of x\, so Xa(^o) ~ 1- Now suppose A e (t')* 
satisfies A ~ —w'qX. Then XA(fcoi^'(7^) = Xx{t)~^ for all t £ T. Taking t = fcg we 
obtain XA(fco) = XA(fco)"^- ^ 

4.8. Theorem. Let X be an integral dominant weight of U . Then the irreducible 
U -module Vx is a {U,a)-module if and only if a+[X) = X and Xxi^o) = Ij where fco 



is as in Lemma 4.7 



In particular, Vx is a ([/, (T)-module if A G a*. It is easy to check that if (J+{X) = X 
the condition xa(^o) ~ 1 does not depend on the choice of the representative fco of 
w'q. 

Proof. Let (V, r) be any ([/, cr)-module and let w be a weight vector in V with 
weight /i. The rule t{uv) ~ a{u)T{v) implies that t(v) is a weight vector with 
weight — (T(/i), and therefore 

kQT(v) is a weight vector with weight aj^{^). (4-3) 
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Now consider the orbit M = UX and the [/-module Q{M) = V\. Assume that 
V\ is a (C/, (7)-module. Let u be a highest-weight vector. Then kQT{v) has weight 
a+(A) according to ( |4.3| ). We conclude from Lemma L7(j) that cr+(A) = A and 



koT{v) = cv for some c g C This implies (fcoT)^(w) = |cpu because r is conjugate 
linear. Hence 

Xx{kl) V = klv = klr^v) = {k^Tf{v) = |c|\, 

where we have used that t commutes with /cq since ko £ K. This implies XA(fco) = 1; 
since |xa(A;o)| = 1- 

Now assume that cr^{X) — A and xaI^o) = 1- Then — (t(A) = WqA, so by Example 



2.9| the orbit M has a well-defined antisymplectic involution t = — cr. Let us check 
that r is antiholomorphic. Because the [/-action is holomorphic and transitive, 
we need only check that (Cko)* ° '''* ■ TxM — > TxAI is conjugate linear, where "£" 
denotes left multiplication on M by elements of U . Since w'q is an involution, 

t{uX) = cr(u)r(A) = (t{u)w'oX = a{u)kg^X. (4.4) 

Let us identi fy T \M with u/ua- Left-multiplying by ko and differentiating at 1 we 



obtain from (L4) the commutative diagram 



Ad(kQ)Q(T 
U > U 



(£fc(,).OT. 



(4.5) 
u/ua > u/ua- 



Consider the root-space decomposition u^ = t"- ©Qg^u^ and select root vectors 
Ea £ u^\ {0} such that E^a ~ —E^ for all a £ R. Denote the complex linear 
extension of Ad(fco) o a hy C. Observe that Cu^ — uf^^ ,., which implies that 
there exist Ca £ C such that 

CEa^ CaE_cr+{a) and CE^a^CaEc,^(a) (4.6) 

for a £ i?+. Let i?'^ = i?:^ U i?^ be the set of all roots which annihilate A. That 
is to say, a £ R^ if and only a^(A) = 0, where a^ denotes the dual root of a. 
Then a+{Rj.) = Rj. because ^^(A) — X and a+ maps i?+ into itself. Hence the Lie 
algebra of the parabolic subgroup Pa is given by 

PA-t^e 0u^e 0u^. 



Therefore 



U^/PA = CE^ 

aeR-\R.t 

TxM ^ u/uA = 

aeR-\Rt 



(4.7) 



where Xq = ^{E-a + Ea) and Yq = ^{E-a — Ea). The complex structure on 
T\M is by definition the puUback of the complex structure on viP/px, so we obtain 



from (4.7) that JXa = Ya and JY^ = -X^. It follows from (M) that for aU 
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CXa = acAg-^^Q) + 0015.+ (a) I 

where Oq — Keca and ba — Imca. Thus CJ = —JC. Together with ( |4.5| ) this 
proves that (Cko)* o r* is conjugate hnear on u/ua. 

Next we show that UX possesses a (U, (7)-equivariant prequantum hne bundle. 
Let L be the homogeneous bundle 

L = U x^>- Cx "^ U^ x^>- Cx, 

where Ca denotes the obvious one-dimensional representation of U\, resp. Pa- This 
bundle has a [/-invariant Hcrmitian structure, namely the one induced by the stan- 
dard inner product on C Put 

tl[u,z] = [a{u)kQ^,z]. 

The assumption that a'+{X) — A implies that tl is well-defined as a map from L to 
itself. The assumption that Xa(^o) ~ ^ implies that it is an involution. It is clear 
that tl is a conjugate linear bundle map and it follows from ([4.4| ) that it is a lifting 



2 



of the involution t on M. The length function on L is given by f{[u,z]) — \z 
which is evidently preserved by r^. One checks directly that tl{uI) = a(u)TL(l), so 
{tl)*£,l = ""(ilf for all £, Eu. Furthermore, {tl)*i^l ^ ~vl and t*4>^ = -$'^(€), so 
by reading ( |4~^ ) backwards we get (TL)*((^M)hor) = (nCA/)!^!- for aU ^. Since M 
is homogeneous, this implies {tl)*'Hi — 'Hr^.H) for all I G L, a nd so t^ preserves the 



connection. Therefore tl is antiholomorphic by Lemma 4.4. The upshot is that L 



is a (C/, (7)-equivariant prequantum line bundle on M. Hence, by Proposition 4.5, 



Vx — Q{M) is a unitary ([/, tT)-module. D 

4.9. Remark (quaternionic modules). If o'+(A) = A but xaC^o) = ~1; then the 
lifting Tl is not an involution, but satisfies rf^u^z] = [u, — 2;]. Otherwise the proof 
goes through as stated and the result is a unitary irreducible [/-module Q{M) with 
an antiholomorphic orthogonal map tq satisfying Tq = — id and tq{uv) = a(u)v. 
In the extreme case where a is the identity such a module is known as a [/-module 
of quaternionic (or symplectic) type, and what we have called a ([/, (T)-module is 
nothing but a [/-module of real (or orthogonal) type. The condition cr-i-(A) = A is 
then equivalent to A = —wqX (wq being the longest element in W), in other words 
to the representation V\ being self-conjugate. 

4.10. Example. If [/ = SU(n) and a is complex conjugation, then W^ = K = 
SO(n) and a = t. Hence every coadjoint orbit is symmetric and every irreducible U- 
module admits the structure of a ([/, cr)-module. On the fundamental representation 
/\ C" the involution is simply complex conjugation with respect to a real form 
/\ W\ On the other hand, if a is the identity, then a — {0} and only a few orbits 
give rise to representations of real type. For instance, /\ C" is self-dual if and only 
if n is even and k — n/2. It is of real type if and only if n = mod 4 and k = n/2. 
(See e.g. Tits @.) 

We can now easily describe the structure of an arbitrary ([/, (T)-module. 

4.11. Addendum. (i) Every finite- dimensional object in the category of (U, a) - 
modules is isomorphic to a direct sum of irreducible objects. 
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(ii) The irreducible objects are 

Vx with cr+(A) 



A and X\{ko) 



V, 



dup 



with cr+(A) ^ X or Xa(^o) 



(4.8) 
(4.9) 



Here A ranges over the dominant weights of U and V^^ "^ stands for the dupli- 



cated module defined in Example ^.15 . 



(iii) As a U -module V^ "^ is isomorphic to Vx 



K 



<T+(A)- 

Note that a module of type ( |4.9| ) can be described as the quantization of the 
disconnected manifold ([/A)'^"p ^ C/A U -a{UX). 



Proof. (i|) is proved by induction on the dimension and (p^) follows from 
Lemma 4.7(|). 



and 



It is clear that the objects (4 



are irreducible and it follows from Theorem 
4.8 that the objects (4.£) are irreducible as well. Now let {V,t) be an arbitrary 
irreducible object. If it is irreducible as a t/-module, we conclude from Theorem 
4.8 that it must be of type (i-i). If not, it must be of the form Vx © t{Vx) = 



Vx © K.+ (A) - V. 



dup 



for some A 

dup 



e A* 



In this case we cannot have cr+(A) = A 

and XA(fco) = 1' ^'^'^ then V^ "^ would be isomorphic to Vx © Va as a {U, cr)-module. 
(The easiest way to see this is by using the bilinear form b defined in (2.5). It 
is not hard to show that if a [/-module E carries two nondegenerate symmetric 
bilinear forms 6i, 62 satisfying bi(uv,a{u)w) = bi{v,w), then /*6i — 62 for some 
[/-equivariant linear map /. Applying this observation to E — V-^ "'^ with 61 being 
the form defined by the involution r and 62 the one defined by the involutions on 
each of the summands Vx, we obtain V^ "^ = Va © V\ as a (C/, cr)-module.) Thus we 
are in case (L9). This proves (P). D 



4.12. Remark. If U is not connected, (|) is still true, but the description of the 
irreducible objects is less clear-cut. However, the fact remains that each such object 
is either of the form V or T/"^"p, where V is irreducible as a [/-module. 

5. The projective case 



In this section we prove Theorem B.l for compact integral Kahler manifolds. 



The gist of the argument is that the Lagrangian Af^ is dense in M for the complex 
Zariski topology. Noteworthy is that the proof applies to singular as wel l as n on- 
singu lar varieties, and that it leads to a stronger conclusion. Corollaries 5.12 and 
5.13 , which involve the dual symmetric pair {G,K). On top of this we obtain a 
simple characterization of the polytope A(M'^) in terms of certain rational weights 
associated with the quantization of M, analogous to the characterization of A(M) 
given by Guillemin and Sternberg |^ and Mumford Q . 

Let (M, to, T, $) be a compact connected Hamiltonian (C/, cr)-manifold equipped 
with a complex structure which is compatible with uj and relative to which t is 
antiholomorphic. Furthermore, as in Proposition 4.5, let L he a holomorphic line 
bundle with a Hermitian connection V such that curv V = —2TTiuj and r lifts to an 
involutive antiholomorphic bundle map tl which preserves V. 



According to Corollary 4.6 we can alternatively think of M as a [/-stable and 
T-stable complex submanifold of W for some unitary ([/, (T)-module V. 

Following Brion H, we start by defining a "rational" analogue of the sets A(M) 
and A(Af'^). Let A^ be the monoid of dominant weights in t^. 
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5.1. Definition. The highest-weight set of M is the subset C{M) of A* ® Q con- 
sisting of all quotients A/n, where A € A^ and n is a positive integer such that the 
irreducible representation V\ occurs in the [/-module r(M, L"). 

Since 

Hom(r(M, L"), Vx) ^ r(M, L") ® r([/A*, La*) = T{M x C/A*, L" H La-), 

we see that £, e C{M) if and only if i^ = A/n for some A G A^ and n > such 
that there exist m Cz M , u Cz U , and a [/-invariant global holomorphic section s of 
L" Kl La* with s(m, uA*) 7^ 0. Here we use the notation A* = —wqX and the fact 
that V* = ^A* • 

5.2. Definition. The highest-weight set of M'^ is the set C{AI'^) consisting of all 
quotients A/n, with A e A^ n a* and n > 0, such that there exist m £ M"^, k £ K, 
and a [/-invariant global holomorphic section s of L" Kl La* with s{m, k\*) ^ 0. 

5.3. Lemma. If AH is nonempty, then C{Ar) = C{M) n a*. 

Proof. The inclusion C(Af^) C C{M) n a* is obvious. Now let ^ € C{M) n a*, write 
^ = A/n with A e A:5_na* and n > 0, and choose a [/-invariant s € r(Mx [/A*, L"K1 
La*) such that s{m,uX*) ^ for some m S M, u G U. We need to show that m 
can be chosen in Af^ and u in /'C. 

Let us first consider the case ^ = 0, where we need merely show that s{m) ^ 



for some m € Af^ . Here the desired result follows from Proposition 2^ , which says 
that M'^ is a totally real Lagrangian: if s vanished on Af^ , then it would vanish on 
all of M by the identity principle. 

The general case is reduced to the special case by means of the "shifting trick" 



one replaces M with the product M x [/A* and L with L" H La*. Lemma 5.4 



below implies that A* G a*, so according to Example ^^ the orbit [/A* has a well- 
defined antisymplectic involution, which we denote here by ta*, and ([/A*)"^^* — 
KX* 7^ by ( ^.6| ). Applying the result of the previous paragraph, we find a point 
in AL^ X {UX*y^* = Af^ x KX* where s does not vanish, i.e. m € Af^ and k e K 
such that s{m, kX*) y^O. D 

5.4. Lemma. Let Wq be the longest element in W. Then WqX = wqX for all 
X <E a*. It follows that wq € W . 

Proof. Let A G a*^. By definition w^ sat isfie s wgA G — a!j^ C — t!j_. Furthermore, 
qA = wA for some w G W^'^ by Lemma B^(y), so wX G — t!J_. At the same time 



w, 



WqX G — t!j^ and therefore wA = wqX, because WX intersects — t!j_ in a unique point. 
We conclude that WqX = wqX. By Lemma B^(|) this implies wo G Afwi<^) = 



W". U 

Let X{M) be the Lie algebra of vector fields on M . Since X{M) has a complex 
structure, the homomorphism ^: u — > A'(Af ) which maps ^ to ^a/ extends naturally 
to a homomorphism of complex Lie algebras, 

A^:u^ — >X{M). 

Recall that the group of biholomorphic transformations of a compact complex man- 
ifold is a complex Lie group. It follows from this that J^ integrates to an action 
of the complex reductive group [/"'. Let tr^ denote the complex linear extension of 
<7 to u^ (and also the holomorphic extension of a to [/^), and let 7u be complex 
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conjugation on u"' relative to u. Put 7^ — u^^xx. Then 7^ is a conjugate linear in- 
volution of u"" because cr^7u = 7uO'''. It is nothing other than complex conjugation 
on u"' relative to the real form 

= «®P, (5.1) 

where p = iq. It follows from ( |2.2D that Aa — tA and therefore 

^S=^-4^ (5-2) 

because r is conjugate linear. Let us write G for the connected real reductive 
subgroup of U^ generated by exp g and Q for the restriction of a'" to G. Then K is 
a maximal compact subgroup of G, d is the Cartan involution, and the symmetric 
pair (g,6) is known as the dual of (u, t); cf. [O, Ch. V]. The following result is 



analogous to Proposition 2.3 



5.5. Proposition. The involution t is G-equivariant. Hence every connected com- 
ponent of Af '^ is G-stable. For every m G M'^ the intersection Urn n A/"^ has a 
finite number of connected components, each of which consists of a single G- orbit. 



Proof. It follows from (5.2) that t^S,m = S,m for all £, £ Q. Consequently T{gm) — 
gT{m) for all g G G and m e M because G is connected. This implies that G 
preserves every component of Af^. That U'^m n M'^ has finitely many connected 
components follows from the fact that it is a real algebraic variety. To prove that 
each component is a single G-orbit it suffices to show that the tangent space to Gm 
at m is equal to TmiW^m) n Tm.M'^ . A general tangent vector to U^m at m is of 
the form Cm = Cm + Cm' where C"*" = ■f^ + iv"^ with ^+, 77+ G t and ^~, 77^ G q. 
According to (|J|, 



.(Cm) = 7b(c)m - (r - "r -r + iv 



M, 



SO Cm is tangent to Af^ if and only if t^{Cm) — Cm if and only if (^ + ir]~^)M = 0, 
i.e. Cm = (C+ + «7?")m G Tra{Gm). D 

5.6. Example (real flag varieties). Let M be a symmetric coadjoint orbit, A/ — 



UX with A G q*. We saw in Example 2.9 that Af^ consists of the single X-orbit 



KX and in the proof of Theorem |4.8| that M has a natural compatible complex 
structure. Therefore G acts in a natural way on M and M'^ consists of a single 
G-orbit. As explained in [^ §5], the stabihzer G\ is the real parabohc subgroup 
i^Aexpiaexpn, where n is the nilpotent algebra g n ®ae-R-\fl' '^^a- Thus Af^ is 



a generalized flag variety of the real reductive group G. See further Example 5.14 
and Section ^. 

We now explain the relationship between highest-weight sets and polytopes. 

5.7. Theorem. Assume that M'^ is nonempty. Then the highest-weight set C{M) 
is equal to the set of rational points in A(Af), and A(Af) is equal to the closure 
of C{M) in t*^. The same statement holds with M replaced by Af^. Therefore 
A(Af '^) = A(A/) n a* and A(Af'^) is a rational convex polytope. 

Proof. The first assertion is proved in |^, ^ ; cf . also [^ . The proof of the second 
assertion follows the same pattern. We begin by proving that C(Af'^) is equal to 
the set of rational points in A(A/^). As a special case we show first that 

OGC(Ar) -^=^ OGA(Ar). (5.3) 
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Assume that G C(Af^). Pick n > and a [/-invariant s e r(M, L") such that 
s(m) 7^ for some m € M'^ . Choose a [/-invariant inner product on u. Then 

grad||<i>(m)f = 2J<i>(77i)^,,,„, (5.4) 

where b : u* ^ u is the hnear isomorphism defined by the inner product, and ^{m)j^j 
is the vector field on M induced by $(m)''. (See e.g. [|l6[ §6].) Let T be the flow 
of the vect or fleld — grad||$||^ and let / be the function (s, s). An easy calculation 
using ( ^.4|) shows that 

j^f{T{t,m))=87T\\^{T{t,m))\\^f{T{t,m))>0. (5.5) 

Let moo = linit^oo-^tC'Ti). Then ( ^.5| ) and the fact that /(m) > imply that 
f{iTiao) > 0. Moreover, limt^oo df[T{t, m)) /dt = and hence 

$(tooo) = 0. (5.6) 



Furthermore, <I>(m) G q* because m G M"^, so i$(m)'' G ?q = p. Together with ( p.4D 
this implies that grad||$(?7i)|p is tangent to the orbit Gm. This orbit is contained in 
M^ by Proposition ^.5| . The conclusion is that the flow JF preserves the submanifold 
M'^ and also the G-orbits inside it. This means that TOqo G Gm C Af^ and hence, 
by ^), G A(Af-). 

Conversely, if G A(M^) then clearly G A(Af), so G C(Af) by the first part 



of this theorem. Hence G C(A/^) by Lemma 5.3. This concludes the proof of 

(O). 

To prove that X/n G C(A/^) if and only if X/n G A(M^), where A G A!^ and 
n > 0, we use the shifting trick as in the proof of Lemma ^.3| . We note that 
X/n G A(Af^) if and only if G A(Af^ x KX*), and that X/n G C(Af^) if and only 
if G C{M^ X KX*). (Here we have multiplied the symplectic form on M by n 
and raised its prequantum line bundle to the n-th power) . Now apply the previous 
argument to the product Af^ x KX*. 

Next we show that C{M'^) is dense in A(Af^). This follows from 



C(Af^) = C{M) n a* = A(Af) n o* D A(Ar), (5.7) 

where in the first equality we have used that the subspace a* of t* is defined over 
the rationals and in the second equality that the closure of C{M) is A(Af). This 

proves the second statement of the theorem. 

The last statement follows immediately from the first two and Lemma p.3\ D 



5.8. Remark. Let a^ be the involution oft* defined by cr+ = —WqO<t. It is easy to 
see that cr_|_ maps A(Af ) into itself, and it follows from Lemma [4.7| (p|) that A(Af ■^) 
is contained in A(Af)'^+. However, if w'q ^ 1, then A(Af)°"+ may be strictly larger 
than A(Af^). 



Theorem 5.7 can be generalized to certain singular subsets of AH . 



5.9. Definition. A {U,a)-pair in Af is a pair {X,Y) of subsets X C M and 
Y C M"^ subject to the following requirements: AT is a [/-stable and r-stable 
irreducible closed complex algebraic subvariety of M such that Areg intersects Af^, 
and Y is the closure of S, where S is any connected component of Areg n Af^. 
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Here X,cg denotes the set of smooth points of X. Standard resuhs in real 
algebraic geometry (see e.g. [g, Ch. 2]) say that Xj-eg n M^ has only finitely many 
connected components and that F is a semi-algebraic subset of M"^ . It follows from 



Proposition 5.5 and the fact that Xiog is t/'^'-stable that S, and hence F, are stable 
under the action of G. It is obvious how to extend the definitions of A and C to this 
setting: one puts A(X) = $(X) n i\ and A(y) = $(y) n t;;.; and in the definition 
of C(M) and C{M'') one replaces M by X, Ar by Y, and L by L\x. 

5.10. Theorem. Let {X,Y) be any {U, a) -pair in M. Then Theorem p.7| remains 
valid if we replace M by X and AH by Y . 



Proof. This is proved in exactly the same way as Theorem 5.7. The hypotheses on 
X and Y imply that Y contains a Lagrangian submanifold of Xj-og. Therefore Y is 
Zariski dense in X, that is to say, the smallest complex subvariety of M containing 
Y is equal to X . This ensures that a section of L|x which vanishes on Y vanishes 
on X. The fact that Y is G-stable implies that the flow T maps Y into itself. 
Since Y is closed the limit of every trajectory in Y is contained in Y . These facts 



guarantee that the arguments used to prove Lemma 5.2 and Theorem 5.7 still work 



in the present situation. D 

For instance, \et X = M and let Y be any connected component of Af^. Then 
(X, Y) is a (C/, (7)-pair. 

5.11. Corollary. IfY is a connected component of AT^ , then A{Y) ~ A(M)na*. 

More interesting examples are obtained by considering any point m in Af "^ and 
letting X be the closure of the orbit [/"'to. Then Xjcg contains [/"'to,, so we can 
take Y to be the closure of any component of U^m n Af^, for instance Y = Gm. 
The conclusion is as follows. 



5.12. Corollary (orbit closures). A(Gm) = A([/"'m) n a* for every m e A/"^. 

This is a generalization to the nonabelian case of Theorem 2 in Atiyah's paper 
|l] . Brion g proved that for "most" m the moment polytope of the orbit closure 
V^m is equal to the full moment polytope A(Af). We show now that a similar 
result holds for the real orbits Gm. This was first pointed out by Duistermaat [D 
in the abelian case. Our proof is patterned on Brion's. If K denotes the field R or 
C, let us refer to open subsets in the Zariski topology over K as K-Zariski open. 

5.13. Corollary. For all {U,(j)-pairs {X,Y) there exists a nonempty C-Zariski 
open subset O of X such that A{Y) = A(Gto) for allme OnY. The setOnY 
is dense in Y . 

The second statement does not follow automatically from the first, because R- 
Zariski open subsets of an irreducible real algebraic variety are not necessarily dense. 
They are always dense, however, if the variety is nonsingular. 

Proof. Let ^i, ^2, ■ • • , & G Q* be the vertices of the polytope A(F). For each i 
select Xi £ A*^ n a* and Ui > such that ^^ = Xi/rii and pick a nonzero [/-invariant 
Si e r(X, i"). Let O be the C-Zariski open subset of X consisting of all to G X 
such that Si{m) ^ for all i. For every to G O n F, Si{m) ^ and therefore 
^i £ C(Gm) for all i. Using Theorem 5.10| we conclude that A(y) C A(Gto,). On 



the other hand, it is plain that A(Gto) C A(y) and so the two polytopes are equal. 
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The subset O intersects the real algebraic variety Xy-cgOM'^ in an M-Zariski open 
subset and it intersects each component S of Xj-^g n M'^ nontrivially since S* is a 
totally real Lagrangian in Xj-^g. Added to the fact that Xi^g D M'^ is nonsingular, 
this implies that OnXregnM"^ is dense in XregflM^. Consequently OnF is dense 
in Y. U 

5.14. Example (products of orbits). Let M = UX x [//i be the product of the 
coadjoint orbits through X, jj, E a*^. Then Af^ = KX x Kfi. Let us assume for 
simplicity that A and /i are generic, so that U\ = U^ = Zk{A) = K' . According to 



Example 5_^ we can identify M'^ with G/B x G/B^ where B is the real Borel sub- 
group K' exp ia exp n of G. We see immediately that the G-orbits on G/B x G/B are 
in a natural one-to-one correspondence with the i?-orbits on G/B. Moreover, this 
correspondence preserves the closure relation among orbits. Applying the Bruhat 
decomposition (Theorem 1.4 in ||l2|, Ch. IX]) we obtain the following result: the 
G-orbits O^ on M"^ are of the form 

0„ = G(i,w) 

for w e W^; and O^ is in the closure of O^ if and only if i; < it;. Here w denotes the 
coset n mod B G G/B determined by any representative n G AfxiA) oi w G W^, 
and "<" denotes the Bruhat-Chevalley ordering on W^. Thus there is exactly 
one closed orbit, the diagonal Oi, and exactly one open orbit, O^"- It is clear 
that A(C'i) = {X + ^J.} and A(0^«) = A(M^). See Section | for a complete 
calculation of A(M'^). Let us here calculate it and its subpoly topes A{Ow) under 
the simplifying assumption that A(M'^) is entirely contained in the interior of V\_. 
Then Theorem pj implies that all vertices of A(M'^) are images under $ of j4-fixed 



points on KX x JC/z. According to Lemma B.4(ii|) the set of A-fixed points is equal 
to W°X X W^, and therefore 

A(Af^) = hull{ uX + vfj.\u, V G W, uX + vfi dominant }. 

Likewise, the vertices of A{Ow) are images of A-fixed points on O^ = Uu<u,^i'- 
An easy computation yields that the A-fixed points on Oy are the points n(l,v) 
with n G 7Va'(A). Thus 

A{Ou]) = hull{ u{X + vfi) \ u, V G W°, V < w, u{X + vfi) dominant }. 

6. The affine case 
Before we treat the general case we prove a version of the Lagrangian convexity 



theorem for affine varieties. This is not really a special case of Theorem B.l , because 



the moment map on an affine variety is not proper, but it is needed in the proof 



of Theorem 3.1. Also, as in the projective case, the result holds for singular as 
well as nonsingular varieties, for instance closures of orbits of the noncompact real 
reductive group G. The idea of the proof is the same as in the projective case. 

Let (y, r) be a (finite-dimensional) unitary ([/, cr)-module and let M he a U- 
stable and r-stable irreducible affine subvariety of V. 

6.1. Definition. The highest-weight set of M is the set C{M) consisting of all 
dominant weights A such that the irreducible representation V\ occurs in the coor- 
dinate ring C[M]. The highest-weight set of Ar is the set C{Ar) of aU A g A^ n a* 
such that there exist m G M, k G K, and a [/-invariant global algebraic section s 
oiLM Lx' with s{rn, kX*) ^ 0. 
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Here L denotes the trivial line bundle on M with the trivial lift of the [/-action. 



The proof of the following result is similar to that of Lemma 5.3 



6.2. Lemma. If M^ contains a smooth point of M , then C{M'^) ~ C{M)r\a* . D 



Proposition 5.5 is valid exactly as stated in the present situation. Here is the 



analogue of Theorem ^ 



6.3. Theorem. Assume that M'^ contains a smooth point of M . T/ien hullQC(Af ) 
is equal to the set of rational points in A(M), and A(A/) is equal to the closure of 
huUjj C(M) in i*^. The same statement holds with M replaced by AH. Therefore 
A(M'^) = A{M) n a* and A{M'^) is a rational convex polyhedral cone. 

Here "hullQ" denotes the convex hull of a subset of a vector space over Q. The 
assumption that M is irreducible implies that C{M) is closed under addition, and 
therefore hullQC(M), resp. hullQC(M^), is the set of all positive rational multiples 
of elements of C(M), resp. C{A-r). 

Proof. See pi, Theorem 4.9] for the first statement. Next, let us show that 
hullQC(M^) is a subset of A(M^). Let C e hullQC(Af^); then A = < G C{M'') 
for some positive integer n. Consider the product M' — M x U\* equipped with 
the symplectic form w' = nui + uj\- and the moment map $' — n$ + t>. . Here 
oj\ denotes the symplectic form on UX and ix the inclusion of UX into u*. We 
need to show that £ A((M')^). This is established by picking a [/-invariant 
global algebraic section s oi L' — L^ Lx- and a point m' = (tti, kX) G {M'Y with 
s{m') ^ 0, and considering the trajectory of m' under the flow T' of — grad|j<i>'|p. 
It is proved in ||2q| th at m'^ — limt_^oo •?^t('Ti') exists. One then shows as in the 



proof of Theorem 5.7 that m'^ G {M'Y and $'(m^) — 0. (One uses the Hermitian 
metric on L defined by the Gaussian exp(— 7rn|j-|p).) 

Now assume that ^ is a rational point in A(M'^). Then ^ e A(Af) n a*, so, by 
the first part of the theorem ^ g hullQC(Af) fl a*, which is equal to hullQC(A//'^) by 



Lemma |6J. We have shown that A(M) n (A* (g) Q) = hullQ C(M^) 



The fact that hullQC(Af^) is dense in A(Af^) is proved as in ( p. 71 ). 

The last assertion of the theorem follows from the first two plus Lemma 6.2. D 



6.4. Corollary. The moment cones A(Af) and A(A/'^) are independent of the 
U XI {±\}-equivariant affine embedding of M into the {U,a)-module V. D 



We leave it to the reader to state and prove analogues of Corollaries 5.12| and 



5.13, and finish this section with a necessary criterion for the origin to be an extreme 



point of the moment cone. 

6.5. Lemma. Assume that A(Af'^) is a proper cone. Then A C [[/, U] Um for all 
m such that U'^m is closed in M . 

Proof. Consider the homogeneous variety X ~ [/"'/[t/'^, C/*^] {/^. The closedness 
of the orbit U'^m ensures that {U'^)m is the complexification of Um and that X is 
affine. According to the proof of pq. Theorem 4.25], A(X) is contained in A(Af) 
and is equal to the kernel of the projection p: u* -^ ([u, u] + Um)*. Therefore, if 
A(Af'^) = A(A/) n a* is a proper cone, then kerp fl a* = {0}. Hence p induces 
an injection a* ^^ ([u, u] -I- u„i)*, and so a C [u, u] -I- Um- This is equivalent to 
A C [[/, U] [/,„. D 
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7. Normal forms 

In this section we extend to the category of Hamiltonian {U, cr)-nianifolds a weU- 
known local normal form theorem of Guillemin and Sternberg ||ll[ and Marie |Q . 
Because this is mostly a routine exercise, which amounts to inserting as and rs in 
the right places, we omit the proof. The techniques involved are a straightforward 
combination of those used in loc. cit., Meyer [E6|, and Duistermaat M. 

Let H he a, closed (but not necessarily connected) cr-stable subgroup of U and 
let y be a finite-dimensional unitary (H, CT)-module, considered as a Hamilton- 
ian {H, a)-manifold with momentum map $y as in Example EJ. According to 



Examples 2.6 and 2.13 the product T*U x F is a Hamiltonian {U x H,a x a)- 



manifold, where U acts on T*U by left multiplication, H acts on T*U by right 



multiplication and linearly on V. Therefore, by Example 2.10, the symplectic quo- 
tient {T*U X V)//H is a Hamiltonian {U, cr)-manifold. There is a naturally defined 
t/-equivariant diffeomorphism between {T*U x V)//H and the associated bundle 

F{H,V) ^U x" {\f X V), 

which depends only on the choice of a connection on the principal iJ-bundle U -^ 
U/H. Here t)" denotes the annihilator of [) in u*. (See e.g. ^, Appendix A].) 

Now let M be an arbitrary Hamiltonian ([/, tT)-manifold and let m e <f>^^(0). 
Let H be the stabilizer of m and V — Tm(Um)'^ /Tm{Um) the symplectic slice at 
m. It is expedient to choose a [/-invariant and cr-anti-invariant compatible almost 
complex structure on M; then the symplectic slice is a unitary representation in a 
natural way. Moreover, if T{m) = m, the subgroup H is cr-stable, the subspacc t)° 
of u* is cr-stable, and V^ is a unitary (H, cr)-module. We can now state the local 
normal form theorem. 

7.1. Theorem (symplectic slices). Let m e $~^(0) ("I M'^ and H = Um- Let V he 
the symplectic slice at m. Then there exists a C/ x {zLl}-stable open neighbourhood of 
m in M which is isomorphic as a Hamiltonian {U, a) -manifold to a neighbourhood 
of the zero section in F(H,V). The isomorphism maps m to [1,0,0]; the moment 
map and the involution on F{H, V) are given by 

t: [u,£,,v] I — > [a{u),~a{£,),Tv{v)], 

respectively, where ry denotes the involution induced on V . D 

One interesting consequence of this result concerns the symplectic quotient M//U. 
The involution r preserves the fibre $^^(0) and maps t/-orbits to t/-orbits. It there- 
fore descends to an involution on M//U, which we denote also by r. If U acts freely 
on $~^(0), then M//U is a symplectic manifold and the involution is antisym- 
plectic. Therefore the fixed-point set {M//UY is a Lagrangian submanifold. The 
Lagrangian quotient or Lagrangian reduced space M'^ //U is by definition the space 
obtained by projecting UAr $'^(0) into M//U. 

7.2. Corollary. IfU acts freely on $"^(0), then the Lagrangian quotient M'^//U 
is a union of connected components of the Lagrangian [M//UY . 

Proof. Clearly both M-" //U and {M//UY are closed and Nr //U C {M//UY, so 
it suffices to show that M'^ jjU is a Lagrangian submanifold. Let m e M"^. We 
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compute in the model F{H, V). Writing f)° = m we have an inclusion 

U" x"' (m-'^ X V) C F{H, vy (7.1) 

of closed Lagrangian submanifolds of F{H, V). Upon taking quotients we obtain 

(V^n<Py\0))/H'' CF{H,Vy//U. (7.2) 

Now we use the assumption that U acts freely. This implies that H = {!}, 



<I>y = 0, and also that the inclusions (7.1) and (7.2) are equalities. Therefore 



F{H,V)//U = V and F{H,Vy//U = V^. The upshot is that F{H,Vy //U is a 
Lagrangian submanifold of F{H, V)//U. D 

7.3. Remark (singular quotients). If we drop the assumption that U acts freely 
on <i>~"'^(0), the symplectic quotient M//U is a sy mplectic stratified space. An ar- 



gument analogous to the proof of Corollary 7.2 shows that the decomposition of 
M l/U into symplectic strata induces a decomposition of the Lagrangian quotient 
M'^ IJU into smooth manifolds, each of which is Lagrangian in the ambient sym- 
plectic stratum of M//U . Moreover, M'^ jjU always intersects the open stratum of 
M//f/, so that dimM^//C/ = idimM//t/. 

Another consequence of the normal form theorem is the fact that M is locally, 
near the orbit through any point m G M'^n<I>~^(0), isomorphic to an affine algebraic 
Hamiltonian (C/, cr)-manifold. As before, let H = Um and let V be the symplectic 
slice at m. Consider the associated bundle U^ x^ V, which is an affine variety 
with a natural algebraic action of U"^. 

7.4. Lemma. There exist a finite- dimensional unitary (U,cr)-module {E,t) and 
a U-equivariant algebraic embedding l: U'" x^ V ^^ E such that the image X = 
l{U X V) is T-stable. Furthermore, if x — i([l, 0]), then ^e{x) = and the sym- 
plectic slice at x [relative to the symplectic structure on X induced by the constant 
symplectic form on E) is isomorphic to V . 

Outline of proof . Without the r, this is |2q. Example 5.5]. Let us show briefly how 
to adapt the argument to the present context. 

We use the fact that for every compact Lie group Q and closed subgroup H there 
exists a finite-dimensional real Cy-module with a vector in it whose stabilizer is equal 
to n. Taking G ^ U x {±1} and H = H x {±1}, where {±1} acts on U and H 
by means of the involution a, gives us a real [/-module i?f that is equipped with 
an involution rf and contains a vector ei such that Ue^ = H, Tj^(ei) — ei, and 
Tf{ue) = a{u)e for all u & U and e € Ef. Let Ei be the complexification of Ef 
and Ti the conjugate linear extension of tJ* to Ei, i.e. the complex linear extension 
composed with complex conjugation. Then (_Ei,ti) is a unitary ([/, cr)-module, 
and the map U — > Ei which sends g to gei descends to a closed equivariant 
algebraic embedding ti : U'^ / H'^ '-^ Ei. The image of ti is ri-stable: if 5 G C/'', 
then g = uexp(i^) with u Cz U and ^ G u, so 

Tiigei) = ri(wexp(z^)ei) = a{ue:xp{-i^))Ti{ei) G U'^ei. 

Moreover, there exists a (complex, finite-dimensional) (C/, ct)- module {E2,T2) to- 
gether with an embedding L2: V ^^ E2 which intertwines the ff-actions and the 
involutions. To prove this, by virtue of the structure theorem. Addendum 



may assume V to be either of type (4.S) or (4.9). (See Remark 4.12 if H is not 



4.11 



connected.) Let F be a (complex, finite-dimensional) (C/, fT)-module containing a 
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vector whose stabilizer is U. Let A be a dominant weight of U. An argument based 
on Peter- Weyl (see e.g. Q) shows that there exists an iJ-equivariant embedding 
j: Vx ^^ E2, where E2 is of the form F(8)---(8)F(8)F*(8)---(8) F*. Observe that 
E2 is a ([/, cr)-module with involution T2 in a natural way. Assume that y = T^ is 
of type ( |4.§| ) . Put t2 = 9 {j + T2 o j o t). Then L2 is iJ-equivariant and intertwines 
the involutions. By Schur's Lemma, L2 is either injective or 0. In the first case we 
are done; in the second case we replace T2 by — T2 and i2 by i(j — T2 o j o t). Now 
assume that V — V^^^ is of type (4.9). Then the intersection oi j{V\) and t(j(Va)) 



is {0}, so t2 = j ® T2 oj o T defines an 7J-equivariant embedding ofV^Vx® Va-^(x) 
into E2 which intertwines r and T2. 

Finally, we embed the associated bundle into Ei E2 by putting i{[g,v]) = 
{gei,gi.2{v)). n 



The upshot of Theorem 7.1 and Lemma 7.4 is as follows 



7.5. Corollary. Under the hypotheses of Theorem 7.1 there exists a U 'A {±1}- 
stable open neighbourhood ofm in M which is isomorphic as a Hamiltonian ([/, cr)- 
manifold to a neighbourhood of x in X . 

8. Proof of the main theorem 

Let M be an arbitrary ([/, o')-manifold and let m £ M'^ . Assume that A = $(m) 
is in the fundamental Weyl chamber. The positive roots of {Ux,T) are by definition 
those positive roots of {U, T) which are perpendicular to A, and the fundamental 
Weyl chamber t^ -^ of Ux is chosen accordingly. Let H — U,n C Ux be the stabilizer 
of m and 

V = T„,{Umr / {T„,{Um) n T„,{Umr) 

the symplectic slice at m. Consider the affine ([/>, (T)-variety X = [/^ x^ V and 
its moment cone A{X) C t^ ;^. 

8.1. Definition. The local moment cone oi M at m is Am = X + A{X). The local 
moment cone of AH at m is AJ^ = A + A(A"^). 



Theorem 3.1 



It follows from Theorem 6.3 that AZ„ — A„, n a*. Here is a local version of 



8.2. Theorem. For every sufficiently small U -stable and r-stable open neighbour- 
hood O of m the set A(0) is a neighbourhood of the vertex A in Am, and A{0'^) is 
a neighbourhood of the vertex A in A^ . 

Proof. The first statement is |2^, Theorem 6.5]. We now prove the seco nd state- 



ment. Under the assumption that A = this follows from Corollary 7.5, together 
with the following observation: if O is a [/-stable and r-stable neighbourhood of 
X in X, then A(0'^) is a neighbourhood of the vertex in A{X'^). This would be 
automatic if the moment map on X were proper, which it usually is not. How- 
ever, we can factor $ into a proper moment map and a linear projection. The 
S'^-action on X defined by complex scalar multiplication on the slice V commutes 
with the [/-action and is Hamiltonian with moment map (f>{[g,v]) = —-t^WvW^. It 
is not hard to show that the joint moment map <I>x0:A'^u*xIRis proper. 
Clearly $ = tt o ($ x (/)), where tt: u* x R ^- u* is the projection onto the first 
factor. For x G X define ^+{x) to be the unique intersection point of the coadjoint 
orbit U^{x) with the Weyl chamber t^j.. Being proper, $+ x (j) maps O^ onto a 
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neighbourhood of the vertex of the U x S'^-moment cone of X'^ and, being linear, 
TT projects this neighbourhood onto a neighbourhood of the vertex in A(0'^). 

The general case can be reduced to the case A = by means of symplectic cross- 
sections. Let s be the (relatively open) face of t!^ containing A. Let ©^ = C/5 -stars, 
where stars stands for the open star lJr>-s ^ oi s and Us is the centralizer of s. Then 
Ss is a maximal slice for the coadjoint action at A. Its preimage M5 = $"^(65) is 
the symplectic cross-section at s. (See e.g. |2^, Section 3.2].) The fact that A G sHa* 
implies that s, and hence stars, are stable under a^. Moreover, the longest element 
w'q G W has a representative ko which is in K' C U^- It follows from this that 
—a{u^) — cr{u)koa+{^) S 65 for all u G U^ and ^ G stars, i.e. 65 is stable under 
~a. Consequently M^ is r-stable and is therefore a Hamiltonian (C/g, cr)-manifold 
in its own right. Moreover, A{Ms) = A(M) n stars and A(MJ) = A{M^) n stars 
by equivariance of the moment map. It therefore suffices to prove the result for AI^ 
instead of M. Since A is annihilated by U^, we can subtract A from the moment 
map to get <I>(m) = 0. D 

Proof of Theorem ^. It is clear that A(M^) is contained in A(M) Ho*. Let A € 
A(M'^). According to pSl Theorem 6.5] the properness of $ implies that the local 
moment cone Am is the same for all m G $^^(A). Hence, by Theor em p.3[ the cone 
A^ is the same for all m G $~^(A) n M"^. Together with Theorem |8.2| this implies 
that a small neighbourhood of A in A(M'^) is equal to a small neighbourhood of A 
in AJJ^. In particular, A(M'^) is locally convex. It is also closed because $ is proper, 
and hence it is globally convex. A convex set is the intersection of all convex cones 
containing it, so 

A(A/r)= fl A;„= fl A™na*C f| A^na* = A(A/)na*, 

and hence A(Af^) = A(M) n a*. This proves (^ 

IM) follows from Lemma 6.5 and Theorem 3.2. D 



9. Applications 

Cotangent bundles. Let S" be a connected manifold equipped with a [/-action 
and an involution r such that t{us) — a-{u)T{s). Let M = T*S be the {U,a)- 
manifold defined in Example |2.6[ The moment map on M is not proper unless S is 
a homogeneous [/-manifold. Nevertheless, Theorem |8.2| enables us to describe the 
images of M and Af"^ — Tg-rS. Assume that S'^ is nonempty and take any point 
s in it. Let H be the stabilizer Us and V^ = TsS/Ts{Us) the shce at s. Then r 
induces an involution t^ on V^ and the symplectic slice F at s is isomorphic to 
the complexification of V'^, equipped with the obvious [/-action and the conjugate 
linear extension of t^. The afhne local model [/"' x-'^ y is isomorphic to the 
complexification of the real afhne [/-variety U x^ V^. 

9.1. Theorem. The set A(T* S) is a rational convex polyhedral cone. It is equal 
to the moment cone of the affine U -variety U x V . Moreover, AiTgrS) is equal 
to A{T*S) n a* = A{U^ x^ V) O a* and is therefore likewise a rational convex 
polyhedral cone. 



Proof. See 28, Theorem 7.6] for the first statement. To prove the second statement, 



observe that the moment map is homogeneous in the cotangent directions and 
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therefore A{Tg^S) is equal to the cone spanned by A(0), where O is any open 
subset of Tg^S containing the zero section. Now apply Theorem g]^. D 

Coadjoint orbits and subgroups. Let M be a coadjoint orbit of U. In g] a 
description was given of the moment polytope of M considered as a Hamiltonian 
manifold for the action of a subgroup of [/. In this section we combine the results 
of 0] with Theorem 3.1 to describe the moment polytope of the real flag variety 



M"^ relative to the action of a subgroup. 

More specifically, let [/ be a closed connected subgroup of U and denote the 
inclusion map C/ — > C/ by /. Assume that U is cr-stable and denote the restriction 



of a to U by a. Then we have a decomposition u = ? q analogous to (B.l). Let 
Q — {ua{u)~^ \ u G U } he the symmetric space and let A be a maximal torus 
in Q containing 1. Let T be a maximal torus in U containing A and let T be a 
maximal torus in U containing T. It is not necessarily possible to choose T in such 
a way that A = T n Q is a, maximal torus in Q, but for simplicity we shall assume 
this to be the case. (It is certainly possible in the examples we discuss below.) Let 
/* : t* ^ t* be the projection induced by /. Then /* maps a* onto a*. 

We assume the orbit M to be symmetric, i.e. M = KX with A £ a*, so that 
M"^ = KX, and we consider AI as a Hamiltonian ([/, (5-)-manifold. The following 



inequalities for A[M'^) are now an immediate consequence of Theorem 3.1 and [pi 
Theorem 3.2.1]. 

9.2. Theorem. Let (A, A) e a+ x a+. Then X e A(M^) if and only if 

w-'^Xef*{w-^X-vC) (9.1) 

for all triples (w, Wjv) G W xW x W^d such that a^ is contained in (j)*(vaii;y). D 

Here W^ci is the relative Weyl set, which is defined as follows. The compatible 
Weyl set Wcom is the set of allw^W such that dim(wt+ nt+) = dimt-f, where t+ 
and t+ are the respective Weyl chambers in t and t. Observe that Wcom is stable 
under left multiplication by Zw{i), the centralizer of t in W. The relative Weyl 
set is then the set of shortest representatives of the left Zw (t)-cosets in Wcom ■ For 
each w, w and v the condition ( |9.1[ ) represents a finite number of linear inequalities. 
Let X = U/T and X = U/T be the flag varieties of U and U, respectively. Then 
<Jw G iJ^'('")(X) denotes the Schubert class associated with w € W and cj): X ^^ X 
the embedding induced by /. We say that a Schubert class a^ is contained in a 
cohomology class c if it appears in the expansion of c in terms of the Schubert basis. 
See 1^ for a more detailed discussion. 

Let A{T^U) Ca'^ X a^^ be the moment cone of the conormal bundle T^U, con- 
sidered as a {/ X [/-manifold. The polytopes A{M'^) can be obtained by slicing this 
cone horizontally. The inequalities (|9.l[ ) also describe A{TtU) and thus Theorem 



3.2 can be regarded as a more explicit version of Theorem 9.1 for S = U 



The maximal torus. The easiest special case of Theorem ^^ is that oi U — T, 
the maximal torus of U . Here of course T = T, W — {1} and /* is the identity 
map. Furthermore, the flag variety X is a point, so ((>* is the trivial homomorphism 
H'{X,Z) — > Z. It follows that cj)*{vawv) 7^ if and only if u = w"^. Hence for 
every A G o!^ the points A in the moment polytope of the real flag variety Af^ are 



described by the inequalities wX G A — C, where w ranges over W. By Lemma B.3 
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this is equivalent to wX G A — C for all w G W. This implies Kostant's result 
A{M'^) — huUiyA, which was discussed in Example 3.4. 



Products of orbits. Let us apply Theorem |9.2| to the diagonal inclusion of a 
group into two copies of itself. We denote the "small" group by U and the "large" 
group by U^. We take the involution on t/^ to be the direct product of a with 
itself; the diagonal f{U) is obviously stable under this involution. The projection 
/* : (t*)^ -^ i* is the addition map and the homomorphism (/)*■. H'[X,'L)®'^ = 
H'{X'^,Z) — > H'{X,Z) is the cup product. Furthermore, the small Weyl chamber 
t+ is contained diagonally in the big chamber t^ and hence Wrc\ = {I}. We obtain 
the following result from Theorem 9.2. 



9.3. Theorem. Let A, /i and ;/ € a!j_. Then v G t^(K\ x K[i) if and only if 



w V < u A 



V jl 



for all u, V and w di W such that a^ is contained in a^ U cr^. 



(9.2) 
D 



Here < denotes the partial ordering on a* defined by the cone C°. There is 
also a more "economical" version of this result, in which the vector inequalities 
( P^ ) are replaced by scalar inequalities, and the conditions on Schubert classes for 
the general flag variety are replaced with conditions on S chub ert classes for the 
Grassmannians of U; cf. |3| Theorem 4.2.1]. See Example 5.14| for a discussion of 
the G-orbits on K\ x Kji. The Ky Fan inequalities of Tam p9| are a special case 
of (U). 

For U = SU(n), a = complex conjugation and K = SO(n) Theorem |9.3| was 
also observed by Fulton as a corollary of a result of Klyachko [|8| . In this case we 
have a = t and so the moment polytope of M is the same as that of Af^. Theorem 
9.3 then represents a complete list of inequalities satisfied by the eigenvalues of a 
sum A + B oi symmetric matrices in terms of the eigenvalues of A and B. Such 
inequalities were first derived by Weyl. See Fulton's paper m for a survey. 



Isotropic flags. We discuss Theorem 9.3 in more detail for U — SU(n), where we 



take the involution to be the inner automorphism defined by 



J^pq — 



-In 







/. 



with p + q = n and p < q. Wc start by giving an explicit description of the varieties 
KX. Observe that U" = K = S(U(p) x U(g)), G = SU(p,g), and that q consists 
of all matrices of the form 

Z^ 
-Z* 



(9.3) 



where Z is a complex p x g-matrix. The action of K is given by 

{C,D)Z ^CZD* (9.4) 

for C G U(p) and D G U(g). The restricted root system is BCp ii p < q and Cp if 
p — q. (See e.g. [^ Ch. VII, §2].) Consider the antidiagonal p x p-matrices 

/ 1\ / 



J = 



1 



V 



1\ 



A = 



A, 



Vi 



J 



(9.5) 
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with Ai, A2, . . . , Ap real. Then a maximal abelian subspace a of q is given by all 
n X n-matrices of the form 



A = 



and a!j_ = a+ is given by Ai > A2 > • • • > Ap > 0. The unitary matrix 





E 



^ b 1' 




diagonalizes a and maps a^ into the standard chamber t^, 

^iJA 
Ad(£;)A= I 

-iAJj 

Now fix A € a^j. and let d — {di,d2, . ■ ■ ,dk) be the partition of p determined by the 
conditions 



Al — • • • — \di > ^di + l — ■ • ■ — Xdi+d2 > ■ ■ ■ > Arfj_| \-dk-i 

Let Td be the manifold of complex flags 

F: Vo CVi C ■ ■■ CV2k+i CV2k+2 
in C" such that Vo = {0}, V2k+2 = C" and 

'd, 
q-p 

d2k+2~i 

Then Ad(_E)A has the same stabilizer as the "standard" flag 



Ap. 



dim Vi — dim Vi-i 



iil<i<k 
iii^k + 1 
iik + 2<i<2k 



Fo : {0} C C^' C 



-<di+d2 



c • • • c CP c C« c €«+'*'= c • • • c CP+«" 



c C". 
UX which 



It follows that there is a unique C/'-'-equivariant isomorphism tp : Td 
sends E~^Fq to A. The flag perpendicular io F £ Td is defined by 

F^-- {0} c y^i+i c y^i c ... c v^i^ c c", 

where the orthogonal complements are taken relative to the nondegenerate Hermit- 
ian form {v,w)r,„ — {EIpqE^^v,w), which has signature q — p. Then F^ G Td, so 



/pq 



_L defines an involution on J-d and we obtain the diagram 



.Frf 



.Frf 



UX 



UX. 



(9.6) 



9.4. Lemma. The diagram ( |9.6| ) commutes. Hence KX is G-equivariantly dif- 
feomorphic to the manifold of flags {0} '^ Vi ^ ■ ■ ■ ^ Vk in C" such that Vi is 
(., ■)pq -isotropic and dim V^i = X^i^i '^j /'-"" i < k. 
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Proof. The first statement follows easily from the fact that Fq — F^ and 

{EIpqE~^Av,AEIpqE^'^w)pq = {v , w) pq 

for all A E U(n). To prove the second statement, recall that KX — {UXy , which 
implies that KX is isomorphic to the variety of self-perpendicular flags in JF^. Now 
observe that if F = F^, then the truncated flag F<k ■ {0} C Vi C • • • C Vj. is 
isotropic and F is completely determined by F<ck- d 



Every matrix of the form (9.3) is in the X-orbit of a unique matrix A in a!j_. The 
numbers Ai > A2 > • ■ • > Ap > in ( |9.5| ) are known as the singular values. Let 
us call the vector (Ai, A2, . . . , Ap) corresponding to a p x g-matrix Z the singular 
spectrum of Z. For A and /i e o!j_ the polytope A{KX x X/i) is the collection of 
singular spectra of all complex matrices of the form A + B, where A and B are 
p X g-matrices with singular spectra A and /i, respectively. (One gets a corresponding 
result for singular values of real p x g-matrices by considering U — SO{p + q) with 
the inner involution given by Ipq. Although the abelian subspace a is the same 
here as in the complex case, the root systems are different and one finds different 
polytopes.) We wish to thank W. Fulton for pointing out the relationship with 
singular values. See also his preprint g, which appeared after the first version of 
this paper. 

In the Lorentzian case p = 1 it is easy to find the inequalities for A(KX x Kfi) 



directly. It is plain from (9.4) or Lemma 9.4 that the orbits of X on q are 2n — 3 



spheres. The restricted Weyl chamber is a half-line, and the quotient map q* -^ R>o 
assigns to a sphere its radius. The moment "polytope" of a product of two spheres 
KX and Kfj, is therefore simply the interval between |A — /i| and A + /i. According 



to Example 5.14, there are two G-orbits on 5*^"""^ x S'^"^'^ corresponding to the 
two elements of W°, namely the diagonal and its complement. The subpolytopes 
associated with these orbits are {A -t- /x}, resp. the whole interval. 



Let us also write out the case p = q = 2. The inequalities (9.2) can be enumerated 
explicitly as explained in S or H . They are of the form 

keK iGX jej 

where TJ^JC range over all triples of ordered subsets of {1,2,3,4} listed in Table 
^ For instance, the triple (1, 2)(2,4)(2,4) encodes the inequality ^2 + 1^4 < Xi + 
X2 + ^i2 + ^J■i. For each triple JJ^IC there is a similar inequality corresponding to 
JXIC. To save space, for each J, J, K, only one of the triples IJJC and JIJC is 
listed in the table. Next to each triple XJK. is listed the dual triple I*J*K.*. (If 
X = (ii < i2 < • • • < ifc) is a fc-tuple of integers between 1 and n, and {1, 2, . . . , 71} \ 
T = (ifc+i < ik+2 < ■ ■ ■ < in) is the complementary n — fc-tuple, then X* is defined 
as {n+l — in, n + l — in-i, . . . ,?i + l — ife+i). The inequality associated with I* J7*/C* 
is obtained from the one associated with JJ^K, by substituting A ^- A*, /x ^- /x*, 
v ^ I'* , where A* = — ujqA.) For those triples which are self-dual we have omitted 
the corresponding entry in the second column. 

We get the inequalities for A(M'^) by substituting Ai -I- A4 = and A2 + A3 = 
(plus similar equalities for /i and ly) into the inequalities for A(M). Most inequalities 
turn out to be redundant. In particular, every inequality becomes identical to its 
dual inequality, and all non-self-dual inequalities with #1 = ^J = #/C = 2 drop 
out. Here is a complete list, consisting of eighteen inequalities (besides the obvious 
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Table 1. Horn-Klyachko inequalities for SU(4) 



triple 


dual triple 


(1)(1)(1) 
(1)(2)(2) 
(1)(3)(3) 
(1)(4)(4) 
(2)(2)(3) 
(2)(3)(4) 


(1,2,3)(1,2,3)(1,2,3) 
(1,2,3)(1,2,4)(1,2,4) 
(1,2,3)(1,3,4)(1,3,4) 
(1,2,3)(2,3,4)(2,3,4) 
(1,2,4)(1,2,4)(1,3,4) 
(1,2,4)(1,3,4)(2,3,4) 



triple 


dual triple 


(1,2)(1,2)(1,2) 
(1,2)(1,3)(1,3) 
(1,2)(1,4)(1,4) 
(1,2)(2,4)(2,4) 
(1,2)(3,4)(3,4) 
(1,3)(1,3)(1,4) 
(1,3)(1,4)(2,4) 
(1,3)(2,4)(3,4) 
(1,4)(1,4)(3,4) 


(1,2)(2,3)(2,3) 

(1,3)(1,3)(2,3) 
(1,3)(2,3)(2,4) 

(2,3)(2,3)(3,4) 



inequalities Ai > A2 > 0, fJ.i > IJ-2 1^ 0, vi > 1^2 ^ 0)'- 



^i 



l/l - 1/2 < 



vi — 1^2 ^ 



i^2 < Ai + A2 + Ml + M2 

Ai + A2 + A^i - M2 
^Ai - A2 + Aii +M2 

-Ai - A2 + Ml - M2 

^Ai - A2 -Ml -M2 

1^1 + 1^2 > |Ai + A2 - Ml - M2I 

J^i + t^2 > |Ai - A2 - Ml + M2|- 

See Figure]^ for an example. Also shown, in dark, is the subpolytope A{Ow), where 
w is the product of the simple reflections si and S2 in W^. The simple roots are 
denoted by ai and 02 and the corresponding fundamental weights by tti and tt2- 

An unso lve d problem. As is obvious from the above examples, the inequalities of 
Theorem 9.2 arc in general highly overdetermined. We believe it should be possible 
to find a complete set of inequalities which uses only data involving the restricted 
Weyl groups and cohomology classes on real flag varieties. 



< Ai + 


Mi 


< 


/Ai 
< 

\A2 


+ M2 

+ Mi 


> 


\A2 


1 +M2 

-Ml 


> 


|Ai- 


-Mil 


> 


IA2- 


-M2I 



\ A + M 




Figure 1. Moment polytopes of KX x Kfj, and O^ "with K 
S(U(2) xU(2)), A=i7ri+7r2,M = 4(7ri+7r2), w ^ S2S1 
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9.5. Problem. Does the statement of Theorem 9.2 remain true if we replace W, 
W, and Wici by W°, W°, and W°^^, respectively, and complex Schubert classes by 
7^/2- Schubert classes on appropriate real flag varieties? 

Here W^ and W^ are the restricted Weyl groups. The restricted relative Weyl 
set W^°i is defined in the same way as Wrci, except that in the definition W is 
replaced by W^, i by a, and t by a. 



In the example of the maximal torus the answer to Problem 9.5 is affirmative. 
In the KX x iiT /^-example it is not very hard to see that the same is true in the 
low-dimensional cases which we wrote out, although in general the question appears 
to be nontrivial. 

Appendix A. Anti-symplectic maps 

The results in this appendix can be found in a somewhat weaker form in the 
papers of Meyer [p6| and Duistermaat 0. Let {V,lj) be a symplectic vector space. 
Recall that a complex structure J on T^ is compatible with lu ii J (z Sp(V, lo) 
and the symmetric bilinear form uj{-, J-) is positive definite. Let 1/(0;) be the set of 
complex structures on V that are compatible with oj and Inn(y) the set of (positive 
definite) inner products on V. Recall that we have maps l: J{uj) — + Inn(y) and 
tt: Inn(V") —t J{ijj) defined by 

i{J){v, w) = uj{v, Jw), 7r{f3) = {~A})-'/^A0, 

where for /? g Inn(V^) the antisymmetric linear map Ap is defined by lu{v,w) = 
(3{Af3V, w). Furthermore, the map l is injective and tt is a left-inverse of t. It follows 
that TT is a deformation retraction and that J^{lli) is contractible. (See McDuff and 
Salamon ||, §2.5].) 

Now let Q he a Lie group with a real character e: tj ^ M^ and suppose that Q 
acts linearly and sesquisymplectically on V in the sense that g*uj — s{g)Lu for g G Q. 
Define an action of Q on J'{llj) by 

9 ■ J = £{9)90 J°9~^- 
Then J is invariant for this action ii J o g = e{g)g o J. Let Q act on Inn(T^) by 
{g ■ /3)(u, w) = Pigv, gw). The proof of the following result is straightforward. 

A.l. Lemma. The maps l andn are Q -equivariant. Hence, ifQ is compact, J{^)'^ 
is contractible and nonempty. D 

Of course, if Q is compact, e takes values in {±1}. Let us denote the kernel of e 
by Gi- 

A. 2. Lemma. Assume that Q is compact. Let g ^ Q be antisymplectic. Then the 
following conditions are equivalent. 

(i) g is an involution. 

(ii) All eigenvalues of g are real. 

(iii) V^ is a Lagrangian subspace. 

If any of these conditions is satisfied, then Q ^ Qi yi {±1} and V^ is totally real 
with respect to any Q -invariant almost complex structure. 

Proof. It is obvious that (|) implies (y). 

Assume that (y) holds. According to Lemma A.l there exists J € J{lo)^ . Then 



gj — —Jg. Moreover, the eigenvalues of g are ±1 because g is orthogonal relative 
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to the inner product i{J). Let A = ±1 be an eigenvalue of g with eigenvector e ^V 
and put f = Je. Then 

gf = gJe = -Jge = -Xf = Tf- (A.l) 

Let W be the real span of e and /. Then Ty is a Hermitian subspace of W. Since g 
is antisymplectic and preserves W, it preserves the symplectic subspace W^ — W^. 
By induction on the dimension we may assume that the fixed points of g\w^ form 
a Lagrangian subspace. On the other hand, it is clear that g\w fixes either e or /, 
depending on the value of A. This implies (|iii|). 

Now suppose that (0) holds. Let {ei, 62, . . . ,e„} be a basis of V^ and put 



fi — Jci. It follows from (A.l) that fi has eigenvalue —1 and therefore 

{ei,/i,e2,/2,-.-,e„,/„} 

is a (symplectic) basis of V . Hence (|) holds. 

If 3 G 5 \ 5i is an involution, the exact sequence Qi ^^ Q -^ {±1} splits and 
hence Q = Qi >i {±1}- The fact that V^ is totally real follows from the following 
trivial observation: let PF be a (real) vector space with a complex structure J and 
let T be an involution which is antiholomorphic in the sense that tJ = —Jr. Then 
the subspaces W^ and W~'^ — JW^ are totally real, t is complex conjugation with 
respect to W^, avidW ^W ®JW^ = [W^)^. U 

Now let (M, Lo) be a connected symplectic manifold and assume that Q acts 
sesquisymplectically on M, i.e. g*LO = e{g)uj for g e Q. Let us denote by J[M,uj) 
the space of almost complex structures on M which are compatible with lo. As 
before, Q acts in a natural way on J'[M,ijj). 

A. 3. Lemma. Assume that Q acts properly on M . 
(i) J{M^uf)'^ is nonempty and connected. 

(ii) // there exist g € G \ Qi and m & M such that g fixes m and has only real 
eigenvalues at to, then g is an involution and M^ is a Lagrangian submanifold. 
Moreover, M^ is totally real with respect to any Q -invariant almost complex 
structure. 



Proof, m follows from Lemma A.l by a standard argument as in |24, §2.6]. (Q) is 



proved by applying Lemma A. 2 to the stabilizer subgroup £/,„ (which is compact 



and contains g) acting on the tangent space TmM. D 

Appendix B. Symmetric pairs 

This appendix is a collection of elementary facts concerning the symmetric pair 
{U,K). In large part, this is well-known material culled from Helgason [IJ] and 
Loose [g2|. Let U he a compact connected Lie group with involution a and let 
K = (C/'")o. Let 

Q = {ua{u)-^ \ueU} 
be the submanifold of symmetric elements. Put q — TiQ. Then Q = expq and the 
map U —>■ Q which sends u to ua{u)~^ induces an isomctry from the symmetric 
space U/U" onto Q. (See e.g. Chapters II and VI in |^3l-) The Lie algebra of U 
decomposes into the ±l-eigenspaces for the involution a, 

u = e©q (B.l) 

and dually we have u* = t* ® q*. Let ^ be a maximal torus (connected flat 
submanifold) in Q containing 1 and let a = TiA. Then a is a maximal abelian 
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subspace of q and A — exp a. Clearly A is stable under a. The fixed-point group 
( "real part" ) is 

A" =A[2] = {a|a2 = f}, 
the subgroup of 2-torsion elements. Hence #A'^ = 2'', where r = dim A, the rank 
of the symmetric space Q. The following assertion is a straightforward corollary of 
the i^Aif-decomposition, Theorem 8.6 in ||l|, Ch. VII]. 

B.l. Lemma. U" := KA[2] = A[2]K . U 

The factors K and A[2\ usually have a nontrivial intersection, so this product 
is rarely semidirect. Now choose a maximal torus T oi U such that T D A. Such 
a torus is always cr-stable. Let K' = Zk{A), the centralizer of A in K. Then 
T' = {K')o n r is a maximal torus of (the identity component of) K' . Moreover, 

T = T'A (B.2) 



and T'nAC A[2]. (See Proposition 3.3 in ||2g, Ch. VI].) From (p. 11) and (|B.2D we 
get the direct-sum decomposition 

t = t' ® a. (B.3) 

Let R be the root system of (C/, T). Since a is an automorphism and t is cr-stable, 
a acts on R. Define 

R' = {aeR\a{a)^a}, R"" = {a\„ \ a <^ R' }. 

Then R' is the set of roots which vanish on a. It is a root subsystem of R, namely 
the root system of {K',T'), and i?" is a (not necessarily reduced) root system in 
a* , known as the system of restricted roots of the symmetric pair (u, t) . We define 
sets of positive roots in R and i?" as follows: fix sets of positive roots R'_^ in R' and 
R^ in E". Then 

R+ = R'+U{aeR\a\ae R^} (B.4) 

is a system of positive roots in R. Let 

t+ = { C e 1 1 a{0 >OioT ae R+} and a+ = { ^ G i | a(e) > for a e i?^ } 

be the associated Weyl chambers in t, resp. a. Then t+ is a fundamental domain 
for the [/-action on u and a+ is a fundamental domain for the /-iT-action on q. Let 
t!j. C t* and a!i_ C a* be the corresponding dual Weyl chambers. Because of the 
way the positive roots in t and a are related we have the following equalities. 

B.2. Lemma. a+ = t+ n q and a!i_ = t!i_ n q*. D 

Likewise, let C be the root cone, i.e. the cone in t* spanned by the positive roots 
i?+, and let C be the restricted root cone, i.e. the cone in a* spanned by R'^. Then 
C is the dual cone of t-)_ and C° is the dual cone of a+. The root cones are related 
as follows. 

B.3. Lemma. C ^ C a* . 

Proof. The inclusion C° C C n a* is obvious. The reverse inclusion follows easily 
from the fact that a(^) = 5 (o^ + o'(q;)) (^) for a e _R and ^ G a. D 

Finally we recall the relationship between the Weyl groups of the root systems 
R, R' and i?", which we denote by W, W and W^, respectively. Observe that a 
acts in a natural way on W because t is cr-stable. 
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B.4. Lemma. (i) W ^ Zw{a), W" = 7VW(a), and W ^ W jW . 
(ii) (C/A)^ n q* = t/A n 0* = T^A n a* = WX for all Ago*. 
(iii) (huUVFA) n 0* = huUVK^A for all X e a* . 

Here "Z" denotes the centralizer of a subalgebra, "A/"" its normalizer, and "hull" 
denotes the convex hull of a subset of a real vector space. 

Proof. The isomorphisms W = Zw{a) and W^ ^ A/'H'(a)/-2M/(a) are in [^, p. 
325]. If w G VF°', then (j{wS,) = (T{w)a{^) = —w^ for all ^ e a, and the refor e 
w € J\fw{<^)- Conversely, if w € J\fw{<^), then w preserves the de com position (B.3). 
(This follows for instance from the fact that the decompositions ( BA ) and ( |B.3| ) are 
orthogonal with respect to the Killing form.) Now let ^ G t and write £, — £,^ + ^~ 
with ^+ € t' and ^- G a. Then 

(t(u')^ — cr(w)(^++^^) = a(uJ<7{^'^)+w(T{^^)^ = (t(w^+— w^^) ~ w^'^+w^^ = w^ 

and hence a{'w) — w. The conclusion is that Afwi^) — W"^ . This proves (|). 

The first equality in M) follows easily from the fact that A is maximal abelian 
in Q. To prove the second equality, observe that UX n a* C UX n t* = WX, so 
UXna* CWXna*. The last equality is Corollary 8.9 of j|l|, Ch. VII]. 

The inclusion hullM^°A C (huUTyA) n a follows from (|i|). The reverse inclusion 
follows from the fact that 

hullVKA= Pi w(A-C) and huUiy°A= f] ui(A-C»), 



weW 



wew^ 



combined with Lemma B.3 



D 



U; a 
U"; K 

Q; q 

u = t® q 
G; e 

A; a 
A[2]; K' 
T'; T = T'A 
t'; t = t'©a 
R', R ; R 
W] W] w 
Wo; w'g; w^ 

t;;a; 

C;C° 

A;;a+ 

7u; 7fl 

(M, u) 
$; r; M^ 

A(M); A(Af^) 



Appendix C. Notation 

compact connected Lie group; involution 

fixed-point group; its identity component 

space of symmetric elements ua{u)^^; its tangent space at 1 

eigenspace decomposition of u under a 

real reductive group dual to U; Cartan involution 

Cartan decomposition of g, where p = iq 

maximal torus in Q; its tangent space at 1 

2-torsion elements in A; centralizer of A in TiT 

maximal torus of K'; maximal torus of U 

Cartan subalgebra of i'; Cartan subalgebra of u 

roots of (u, t); roots of (6', t'); restricted roots 

Weyl groups of R; R'; R° 

longest elements of W] W; W°- 

fundamental Weyl chambers for R\ resp. i?° 

cone spanned by i?+; resp. R\ 

set of dominant weights; involution on t^ induced by a 

irreducible [/-module with highest weight A G A^ 

complex conjugations on u"' relative to u; resp. g 

symplectic manifold acted on by V 

moment map; antisymplectic involution; fixed-point set 

vector field induced by ^ G u; component of $ along ^ 

moment polytopes <i>(A/) n t;^; resp. $(1/"") n t;^ 



moment maps and riemannian symmetric pairs 35 

References 

1. M. Atiyah, Convexity and commuting Hamiltonians, Bull. London Math. Soc. 14 (1982), 
no. 1, 1-15. 

2. R. Benedetti and J. -J. Risler, Real algebraic and semi- algebraic sets, Hermann, Paris, 1990. 

3. A. Berenstein and R. Sjamaar, Coadjoint o rbits, moment noly toves. and the Hilbert- Mumford 



criterion, J. Amer. Math. Soc, to appear, math.SG/9810125 



4. G. Bredon, Introduction to compact transformation groups, Pure and Applied Mathematics, 
vol. 46, Academic Press, New York, 1972. 

5. M. Brion, Sur I'image de I'application moment, Seminaire d'algebre Paul Dubreil et Marie- 
Paule Malliavin (Paris, 1986) (M.-P. Malliavin, ed.). Lecture Notes in Mathematics, vol. 1296, 
Springer- Verlag, Berlin, 1987, pp. 177-192. 

6. J. J. Duistermaat, Convexity and tightness for restrictions of Hamiltonian functions to fixed 
point sets of an antisymplectic involution. Trans. Amer. Math. Soc. 275 (1983), no. 1, 417- 
429. 

7. W. Fulton, Eigenvalues of sums of Hermitian matrices (after A. Klyachko), Seminaire 
Bourbaki, 50eme annee, expose no. 845 (Paris, 1997-98), Asterisque, vol. 252, Societe 
Mathematique de France, Paris, 1998, pp. 255-269. 

8. , Eigenvalues, invariant factors, highest weights, and Schubert calculus, preprint. Uni- 
versity of Michigan, 1999. 

9. V. Guillemin and S. Sternberg, Convexity properties of the moment mapping. Invent. Math. 
67 (1982), no. 3, 491-513. 

10. , Geometric quantization and multiplicities of group representations. Invent. Math. 67 

(1982), no. 3, 515-538. 

11. , A normal form for the moment map. Differential Geometric Methods in Mathemat- 
ical Physics (Jerusalem, 1982), Reidel, Dordrecht, 1984, pp. 161-175. 

12. S. Helgason, Differential geometry, Lie groups, and symmetric spaces. Academic Press, New 
York, 1978. 

13. J. Hilgert, K.-H. Neeb, and W. Plank, Symplectic convexity theorems and coadjoint orbits, 
Compositio Math. 94 (1994), no. 2, 129-180. 

14. K. Janich, Differenzierbare G-Mannigfaltigkeiten, Lecture Notes in Mathematics, vol. 59, 
Springer- Verlag, Berlin, 1968. 

15. J. Jurkiewicz, Torus embeddings, polyhedra, k* -actions and homology, Dissertationcs Math. 
(Rozprawy Mat.) 236 (1985), 64. 

16. F. Kirwan, Cohomology of quotients in symplectic and algebraic geometry. Mathematical 
Notes, vol. 31, Princeton University Press, Princeton, N.J., 1984. 

17. , Convexity properties of the moment mapping. III, Invent. Math. 77 (1984), no. 3, 

547-552. 

18. A. Klyachko, Stable bundles, representation theory and Hermitian operators, Selecta Math. 
(N.S.) 4 (1998), no. 3, 419-445. 

19. B. Kostant, Quantization and unitary representations. I. Prequantization, Lectures in Mod- 
ern Analysis and Applications, III (Washington, D.C.) (C. T. Taam, ed.). Lecture Notes in 
Mathematics, vol. 170, Springer- Verlag, Berlin, 1970, pp. 87-208. 

20. , On convexity, the Weyl group and the Iwasawa decomposition, Ann. Sci. Ecole Norm. 

Sup. (4) 6 (1973), 413-455. 

21. E. Lerman, E. Meinrenken, S. Tolman, and C. Woodward, Nonabelian convexity by symplectic 
cuts. Topology 37 (1998), no. 2, 245-259. 

22. O. Loos, Symmetric spaces, vols. I and II, W. A. Benjamin, Inc., New York-Amsterdam, 1969. 

23. C.-M. Marie, Modele d'action hamiltonienne d'un groupe de Lie sur une variete symplectique. 
Rend. Sem. Mat. Univ. Politec. Torino 43 (1985), no. 2, 227-251. 

24. D. McDuff and D. Salamon, Introduction to symplectic topology, Oxford Mathematical Mono- 
graphs, Oxford University Press, New York, 1995. 

25. E. Meinrenken and R. Sjamaar, Singular reduction and quantization. Topology 38 (1999), 
no. 4, 699-762. 

26. K. Meyer, Hamiltonian systems with a discrete symmetry, J. Differential Equations 41 (1981), 
no. 2, 228-238. 

27. L. Ness, A stratification of the null cone via the moment map, Amer. J. Math. 106 (1984), 
no. 6, 1281-1329, with an appendix by D. Mumford. 



36 LUIS O'SHEA AND REYER SJAMAAR 

28. R. Sjainaar, Convexity properties of the moment mapping re-examined, Adv. Math. 138 
(1998), no. 1, 46-91. 

29. T.-Y. Tarn, A unified extension of two results of Ky Fan on the sum of matrices, Proc. Amcr. 
Math. Soc. 126 (1998), no. 9, 2607-2614. 

30. J. Tits, Tabellen zu den einfachen Lie Gruppen und ihren Darstellungen, Lecture Notes in 
Mathematics, vol. 40, Springer- Verlag, BerUn, 1967. 

Department of Mathematics, Cornell University, Ithaca, New York 14853-7901 
E-mail address: lulsOmath. cornell.edu 

E-mail address: sjainaaramath.cornell.edu 



